2023-09-30T01:38:35.905 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-30T01:38:35.906 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407090 --description rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/one workloads/rados_mon_workunits} --owner scheduled_yuriw@teuthology --name yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407090/orig.config.yaml 2023-09-30T01:38:35.942 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T01:38:36.025 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407090 branch: wip-yuri4-testing-2023-09-29-0928-quincy description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/one workloads/rados_mon_workunits} email: yweinste@redhat.com first_in_suite: false job_id: '7407090' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri4-testing-2023-09-29-0928-quincy ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - but it is still running - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(AUTH_BAD_CAPS\) sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-09-29-0928-quincy sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 9810 sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-09-29-0928-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL4Ne8rnE0fKgnLxndlIRGawJ6kI6jVZPJd3ia5X2z5siGW7NsynibbkY3Sa/LPpAgVwpTy1xB+DLeEy9osBQQA= smithi050.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPKVvvKVeE2oRIix/AIkKakF3O4QIvkSEqHCqfZMbbIAoWcstU9WTGbXpT9G0dSp1ejGKy6wHwNz/gMXjOyAK9U= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 20 thrash_delay: 1 - workunit: clients: client.0: - mon/pool_ops.sh - mon/crush_ops.sh - mon/osd.sh - mon/caps.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-29_19:38:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-30T01:38:36.025 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa; will attempt to use it 2023-09-30T01:38:36.026 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks 2023-09-30T01:38:36.026 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-30T01:38:36.027 INFO:teuthology.task.internal:Checking packages... 2023-09-30T01:38:36.050 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '93dce5e17918f8aca65993a7a2dc40233a47b481' 2023-09-30T01:38:36.050 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-30T01:38:36.050 INFO:teuthology.packaging:ref: None 2023-09-30T01:38:36.050 INFO:teuthology.packaging:tag: None 2023-09-30T01:38:36.051 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T01:38:36.051 INFO:teuthology.packaging:sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:38:36.051 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T01:38:36.210 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1165.g93dce5e1 2023-09-30T01:38:36.210 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-30T01:38:36.217 INFO:teuthology.task.internal:no buildpackages task found 2023-09-30T01:38:36.217 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-30T01:38:36.233 INFO:teuthology.task.internal:Saving configuration 2023-09-30T01:38:36.247 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-30T01:38:36.253 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T01:38:36.277 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407090', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 01:33:51.626510', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL4Ne8rnE0fKgnLxndlIRGawJ6kI6jVZPJd3ia5X2z5siGW7NsynibbkY3Sa/LPpAgVwpTy1xB+DLeEy9osBQQA='} 2023-09-30T01:38:36.301 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407090', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 01:33:51.628404', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPKVvvKVeE2oRIix/AIkKakF3O4QIvkSEqHCqfZMbbIAoWcstU9WTGbXpT9G0dSp1ejGKy6wHwNz/gMXjOyAK9U='} 2023-09-30T01:38:36.302 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-30T01:38:36.307 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2023-09-30T01:38:36.308 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2023-09-30T01:38:36.308 INFO:teuthology.run_tasks:Running task console_log... 2023-09-30T01:38:36.427 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9babc05ee0>, signals=[15]) 2023-09-30T01:38:36.427 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-30T01:38:36.433 INFO:teuthology.task.internal:Opening connections... 2023-09-30T01:38:36.433 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2023-09-30T01:38:36.434 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:38:36.501 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2023-09-30T01:38:36.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:38:36.567 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-30T01:38:36.573 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2023-09-30T01:38:36.591 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2023-09-30T01:38:36.592 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2023-09-30T01:38:36.649 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="Red Hat Enterprise Linux" 2023-09-30T01:38:36.649 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="8.4 (Ootpa)" 2023-09-30T01:38:36.650 INFO:teuthology.orchestra.run.smithi007.stdout:ID="rhel" 2023-09-30T01:38:36.650 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="fedora" 2023-09-30T01:38:36.650 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="8.4" 2023-09-30T01:38:36.650 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el8" 2023-09-30T01:38:36.650 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-09-30T01:38:36.650 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31" 2023-09-30T01:38:36.651 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-09-30T01:38:36.651 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://www.redhat.com/" 2023-09-30T01:38:36.651 INFO:teuthology.orchestra.run.smithi007.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-09-30T01:38:36.651 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-30T01:38:36.651 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:38:36.651 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-30T01:38:36.652 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-09-30T01:38:36.652 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-09-30T01:38:36.652 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-09-30T01:38:36.653 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2023-09-30T01:38:36.677 DEBUG:teuthology.orchestra.run.smithi050:> uname -m 2023-09-30T01:38:36.695 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2023-09-30T01:38:36.696 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2023-09-30T01:38:36.752 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="Red Hat Enterprise Linux" 2023-09-30T01:38:36.753 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="8.4 (Ootpa)" 2023-09-30T01:38:36.753 INFO:teuthology.orchestra.run.smithi050.stdout:ID="rhel" 2023-09-30T01:38:36.753 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE="fedora" 2023-09-30T01:38:36.753 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="8.4" 2023-09-30T01:38:36.753 INFO:teuthology.orchestra.run.smithi050.stdout:PLATFORM_ID="platform:el8" 2023-09-30T01:38:36.753 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-09-30T01:38:36.753 INFO:teuthology.orchestra.run.smithi050.stdout:ANSI_COLOR="0;31" 2023-09-30T01:38:36.754 INFO:teuthology.orchestra.run.smithi050.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-09-30T01:38:36.754 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://www.redhat.com/" 2023-09-30T01:38:36.754 INFO:teuthology.orchestra.run.smithi050.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-09-30T01:38:36.754 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-30T01:38:36.754 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:38:36.754 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-30T01:38:36.754 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-09-30T01:38:36.754 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-09-30T01:38:36.755 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-09-30T01:38:36.755 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2023-09-30T01:38:36.777 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-30T01:38:36.786 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-30T01:38:36.791 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-30T01:38:36.791 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2023-09-30T01:38:36.793 DEBUG:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2023-09-30T01:38:36.810 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-30T01:38:36.821 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-30T01:38:36.821 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2023-09-30T01:38:36.850 DEBUG:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2023-09-30T01:38:36.865 INFO:teuthology.orchestra.run.smithi007.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-09-30T01:38:36.868 INFO:teuthology.orchestra.run.smithi050.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-09-30T01:38:36.868 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-30T01:38:36.927 INFO:teuthology.run_tasks:Running task kernel... 2023-09-30T01:38:36.941 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-30T01:38:36.941 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-30T01:38:36.941 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T01:38:36.942 DEBUG:teuthology.orchestra.run.smithi050:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T01:38:36.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:38:36.958 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2023-09-30T01:38:36.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:38:36.959 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2023-09-30T01:38:37.015 INFO:teuthology.orchestra.run.smithi007.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-09-30T01:38:37.016 INFO:teuthology.task.kernel:Running kernel on smithi007: 4.18.0-305.25.1.el8_4.x86_64 2023-09-30T01:38:37.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum install -y kernel 2023-09-30T01:38:37.017 INFO:teuthology.orchestra.run.smithi050.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-09-30T01:38:37.017 INFO:teuthology.task.kernel:Running kernel on smithi050: 4.18.0-305.25.1.el8_4.x86_64 2023-09-30T01:38:37.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum install -y kernel 2023-09-30T01:38:38.470 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:38:38.471 INFO:teuthology.orchestra.run.smithi050.stdout:Unable to read consumer identity 2023-09-30T01:38:38.564 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:38:38.564 INFO:teuthology.orchestra.run.smithi007.stdout:Unable to read consumer identity 2023-09-30T01:38:39.879 INFO:teuthology.orchestra.run.smithi050.stdout:Copr repo for python3-asyncssh owned by ceph 31 kB/s | 3.5 kB 00:00 2023-09-30T01:38:39.978 INFO:teuthology.orchestra.run.smithi007.stdout:Copr repo for python3-asyncssh owned by ceph 37 kB/s | 3.5 kB 00:00 2023-09-30T01:38:40.496 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 28 MB/s | 16 MB 00:00 2023-09-30T01:38:41.132 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-09-30T01:38:43.974 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 4.6 MB/s | 24 kB 00:00 2023-09-30T01:38:44.555 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 4.6 MB/s | 24 kB 00:00 2023-09-30T01:38:44.746 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-09-30T01:38:44.746 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-09-30T01:38:44.747 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-09-30T01:38:44.898 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:38:44.899 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-09-30T01:38:44.899 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:38:45.011 DEBUG:teuthology.orchestra.run.smithi050:> echo no | sudo yum reinstall kernel || true 2023-09-30T01:38:45.307 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-09-30T01:38:45.307 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-09-30T01:38:45.307 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-09-30T01:38:45.376 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:38:45.376 INFO:teuthology.orchestra.run.smithi050.stdout:Unable to read consumer identity 2023-09-30T01:38:45.461 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:38:45.462 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-09-30T01:38:45.462 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:38:45.588 DEBUG:teuthology.orchestra.run.smithi007:> echo no | sudo yum reinstall kernel || true 2023-09-30T01:38:45.687 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:02 ago on Sat 30 Sep 2023 01:38:43 AM UTC. 2023-09-30T01:38:45.830 INFO:teuthology.orchestra.run.smithi050.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-09-30T01:38:45.830 INFO:teuthology.orchestra.run.smithi050.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T01:38:45.830 INFO:teuthology.orchestra.run.smithi050.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T01:38:45.854 INFO:teuthology.orchestra.run.smithi050.stderr:Error: No packages marked for reinstall. 2023-09-30T01:38:45.904 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum reinstall -y kernel || true 2023-09-30T01:38:45.951 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:38:45.951 INFO:teuthology.orchestra.run.smithi007.stdout:Unable to read consumer identity 2023-09-30T01:38:46.264 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:02 ago on Sat 30 Sep 2023 01:38:44 AM UTC. 2023-09-30T01:38:46.268 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:38:46.268 INFO:teuthology.orchestra.run.smithi050.stdout:Unable to read consumer identity 2023-09-30T01:38:46.410 INFO:teuthology.orchestra.run.smithi007.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-09-30T01:38:46.410 INFO:teuthology.orchestra.run.smithi007.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T01:38:46.411 INFO:teuthology.orchestra.run.smithi007.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T01:38:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:Error: No packages marked for reinstall. 2023-09-30T01:38:46.492 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum reinstall -y kernel || true 2023-09-30T01:38:46.575 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:03 ago on Sat 30 Sep 2023 01:38:43 AM UTC. 2023-09-30T01:38:46.737 INFO:teuthology.orchestra.run.smithi050.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-09-30T01:38:46.737 INFO:teuthology.orchestra.run.smithi050.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T01:38:46.737 INFO:teuthology.orchestra.run.smithi050.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T01:38:46.751 INFO:teuthology.orchestra.run.smithi050.stderr:Error: No packages marked for reinstall. 2023-09-30T01:38:46.801 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T01:38:46.854 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:38:46.855 INFO:teuthology.orchestra.run.smithi007.stdout:Unable to read consumer identity 2023-09-30T01:38:46.932 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-09-30T01:38:46.932 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-09-30T01:38:46.933 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-09-30T01:38:46.933 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T01:38:46.933 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T01:38:46.933 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-09-30T01:38:46.933 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi050.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T01:38:46.933 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T01:38:47.063 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-09-30T01:38:47.063 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-09-30T01:38:47.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo rpm -qi grub2-tools 2023-09-30T01:38:47.092 INFO:teuthology.orchestra.run.smithi050.stdout:Name : grub2-tools 2023-09-30T01:38:47.092 INFO:teuthology.orchestra.run.smithi050.stdout:Epoch : 1 2023-09-30T01:38:47.092 INFO:teuthology.orchestra.run.smithi050.stdout:Version : 2.02 2023-09-30T01:38:47.092 INFO:teuthology.orchestra.run.smithi050.stdout:Release : 99.el8_4.1 2023-09-30T01:38:47.092 INFO:teuthology.orchestra.run.smithi050.stdout:Architecture: x86_64 2023-09-30T01:38:47.092 INFO:teuthology.orchestra.run.smithi050.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-09-30T01:38:47.093 INFO:teuthology.orchestra.run.smithi050.stdout:Group : System Environment/Base 2023-09-30T01:38:47.093 INFO:teuthology.orchestra.run.smithi050.stdout:Size : 9264631 2023-09-30T01:38:47.093 INFO:teuthology.orchestra.run.smithi050.stdout:License : GPLv3+ 2023-09-30T01:38:47.093 INFO:teuthology.orchestra.run.smithi050.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-09-30T01:38:47.093 INFO:teuthology.orchestra.run.smithi050.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-09-30T01:38:47.093 INFO:teuthology.orchestra.run.smithi050.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-09-30T01:38:47.093 INFO:teuthology.orchestra.run.smithi050.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-09-30T01:38:47.093 INFO:teuthology.orchestra.run.smithi050.stdout:Relocations : (not relocatable) 2023-09-30T01:38:47.093 INFO:teuthology.orchestra.run.smithi050.stdout:Packager : Red Hat, Inc. 2023-09-30T01:38:47.093 INFO:teuthology.orchestra.run.smithi050.stdout:Vendor : Red Hat, Inc. 2023-09-30T01:38:47.094 INFO:teuthology.orchestra.run.smithi050.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-30T01:38:47.094 INFO:teuthology.orchestra.run.smithi050.stdout:Summary : Support tools for GRUB. 2023-09-30T01:38:47.094 INFO:teuthology.orchestra.run.smithi050.stdout:Description : 2023-09-30T01:38:47.094 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:38:47.094 INFO:teuthology.orchestra.run.smithi050.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-30T01:38:47.094 INFO:teuthology.orchestra.run.smithi050.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-30T01:38:47.094 INFO:teuthology.orchestra.run.smithi050.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-30T01:38:47.094 INFO:teuthology.orchestra.run.smithi050.stdout:hardware devices. 2023-09-30T01:38:47.094 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:38:47.094 INFO:teuthology.orchestra.run.smithi050.stdout:This subpackage provides tools for support of all platforms. 2023-09-30T01:38:47.095 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-30T01:38:47.095 INFO:teuthology.task.kernel:Updating grub on smithi050 to boot 4.18.0-425.3.1.el8.x86_64 2023-09-30T01:38:47.096 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-30T01:38:47.158 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:03 ago on Sat 30 Sep 2023 01:38:44 AM UTC. 2023-09-30T01:38:47.301 INFO:teuthology.orchestra.run.smithi007.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-09-30T01:38:47.301 INFO:teuthology.orchestra.run.smithi007.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T01:38:47.301 INFO:teuthology.orchestra.run.smithi007.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-09-30T01:38:47.320 INFO:teuthology.orchestra.run.smithi007.stderr:Error: No packages marked for reinstall. 2023-09-30T01:38:47.372 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T01:38:47.502 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-09-30T01:38:47.502 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-09-30T01:38:47.502 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-09-30T01:38:47.502 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T01:38:47.503 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T01:38:47.503 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-09-30T01:38:47.503 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T01:38:47.503 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T01:38:47.633 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-09-30T01:38:47.633 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-09-30T01:38:47.634 DEBUG:teuthology.orchestra.run.smithi007:> sudo rpm -qi grub2-tools 2023-09-30T01:38:47.661 INFO:teuthology.orchestra.run.smithi007.stdout:Name : grub2-tools 2023-09-30T01:38:47.662 INFO:teuthology.orchestra.run.smithi007.stdout:Epoch : 1 2023-09-30T01:38:47.662 INFO:teuthology.orchestra.run.smithi007.stdout:Version : 2.02 2023-09-30T01:38:47.662 INFO:teuthology.orchestra.run.smithi007.stdout:Release : 99.el8_4.1 2023-09-30T01:38:47.662 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: x86_64 2023-09-30T01:38:47.662 INFO:teuthology.orchestra.run.smithi007.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-09-30T01:38:47.662 INFO:teuthology.orchestra.run.smithi007.stdout:Group : System Environment/Base 2023-09-30T01:38:47.662 INFO:teuthology.orchestra.run.smithi007.stdout:Size : 9264631 2023-09-30T01:38:47.662 INFO:teuthology.orchestra.run.smithi007.stdout:License : GPLv3+ 2023-09-30T01:38:47.662 INFO:teuthology.orchestra.run.smithi007.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-09-30T01:38:47.663 INFO:teuthology.orchestra.run.smithi007.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-09-30T01:38:47.663 INFO:teuthology.orchestra.run.smithi007.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-09-30T01:38:47.663 INFO:teuthology.orchestra.run.smithi007.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-09-30T01:38:47.663 INFO:teuthology.orchestra.run.smithi007.stdout:Relocations : (not relocatable) 2023-09-30T01:38:47.663 INFO:teuthology.orchestra.run.smithi007.stdout:Packager : Red Hat, Inc. 2023-09-30T01:38:47.663 INFO:teuthology.orchestra.run.smithi007.stdout:Vendor : Red Hat, Inc. 2023-09-30T01:38:47.663 INFO:teuthology.orchestra.run.smithi007.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-30T01:38:47.663 INFO:teuthology.orchestra.run.smithi007.stdout:Summary : Support tools for GRUB. 2023-09-30T01:38:47.663 INFO:teuthology.orchestra.run.smithi007.stdout:Description : 2023-09-30T01:38:47.663 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:38:47.663 INFO:teuthology.orchestra.run.smithi007.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-30T01:38:47.663 INFO:teuthology.orchestra.run.smithi007.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-30T01:38:47.663 INFO:teuthology.orchestra.run.smithi007.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-30T01:38:47.664 INFO:teuthology.orchestra.run.smithi007.stdout:hardware devices. 2023-09-30T01:38:47.664 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:38:47.664 INFO:teuthology.orchestra.run.smithi007.stdout:This subpackage provides tools for support of all platforms. 2023-09-30T01:38:47.665 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-30T01:38:47.665 INFO:teuthology.task.kernel:Updating grub on smithi007 to boot 4.18.0-425.3.1.el8.x86_64 2023-09-30T01:38:47.665 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-30T01:38:48.608 INFO:teuthology.orchestra.run.smithi050.stderr:Generating grub configuration file ... 2023-09-30T01:38:49.119 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2023-09-30T01:38:50.729 INFO:teuthology.orchestra.run.smithi050.stderr:done 2023-09-30T01:38:50.732 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2023-09-30T01:38:50.746 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.egbfhb0w6s 2023-09-30T01:38:50.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.egbfhb0w6s 2023-09-30T01:38:50.856 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.egbfhb0w6s 2023-09-30T01:38:50.973 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.egbfhb0w6s is 6KB 2023-09-30T01:38:51.021 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.egbfhb0w6s 2023-09-30T01:38:51.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo /bin/ls /boot/loader/entries || true 2023-09-30T01:38:51.106 INFO:teuthology.orchestra.run.smithi050.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-09-30T01:38:51.106 INFO:teuthology.orchestra.run.smithi050.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-09-30T01:38:51.106 INFO:teuthology.orchestra.run.smithi050.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-09-30T01:38:51.106 INFO:teuthology.orchestra.run.smithi050.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-09-30T01:38:51.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-09-30T01:38:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:done 2023-09-30T01:38:51.193 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2023-09-30T01:38:51.207 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.OaPIDQ9rid 2023-09-30T01:38:51.207 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.OaPIDQ9rid 2023-09-30T01:38:51.251 DEBUG:teuthology.orchestra.run.smithi050:> sudo shutdown -r now 2023-09-30T01:38:51.292 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.OaPIDQ9rid 2023-09-30T01:38:51.410 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.OaPIDQ9rid is 6KB 2023-09-30T01:38:51.458 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.OaPIDQ9rid 2023-09-30T01:38:51.472 DEBUG:teuthology.orchestra.run.smithi007:> sudo /bin/ls /boot/loader/entries || true 2023-09-30T01:38:51.534 INFO:teuthology.orchestra.run.smithi007.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-09-30T01:38:51.534 INFO:teuthology.orchestra.run.smithi007.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-09-30T01:38:51.534 INFO:teuthology.orchestra.run.smithi007.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-09-30T01:38:51.534 INFO:teuthology.orchestra.run.smithi007.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-09-30T01:38:51.535 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-09-30T01:38:51.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2023-09-30T01:39:21.282 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-09-30T01:39:21.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-09-30T01:39:21.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:39:21.723 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-09-30T01:39:21.723 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-09-30T01:39:21.724 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:39:39.770 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.50 2023-09-30T01:39:40.286 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.7 2023-09-30T01:39:48.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-09-30T01:39:48.779 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:39:49.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-09-30T01:39:49.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:40:48.838 DEBUG:teuthology.orchestra.remote:timed out 2023-09-30T01:40:49.290 DEBUG:teuthology.orchestra.remote:timed out 2023-09-30T01:41:00.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-09-30T01:41:00.851 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:41:01.139 DEBUG:teuthology.orchestra.run.smithi050:> true 2023-09-30T01:41:01.292 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-09-30T01:41:01.292 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:41:01.389 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-09-30T01:41:01.389 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-425.3.1.el8.x86_64"... 2023-09-30T01:41:01.389 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2023-09-30T01:41:01.405 INFO:teuthology.orchestra.run.smithi050.stdout:4.18.0-425.3.1.el8.x86_64 2023-09-30T01:41:01.405 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-09-30T01:41:01.406 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T01:41:01.406 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-09-30T01:41:01.629 DEBUG:teuthology.orchestra.run.smithi007:> true 2023-09-30T01:41:01.880 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-09-30T01:41:01.880 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-425.3.1.el8.x86_64"... 2023-09-30T01:41:01.881 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2023-09-30T01:41:01.896 INFO:teuthology.orchestra.run.smithi007.stdout:4.18.0-425.3.1.el8.x86_64 2023-09-30T01:41:01.896 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-09-30T01:41:01.897 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T01:41:01.897 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-09-30T01:41:02.407 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-09-30T01:41:02.407 DEBUG:teuthology.orchestra.run.smithi050:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T01:41:02.512 INFO:teuthology.orchestra.run.smithi050.stdout:ttyS1 2023-09-30T01:41:02.525 DEBUG:teuthology.parallel:result is None 2023-09-30T01:41:02.898 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-09-30T01:41:02.898 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T01:41:02.977 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2023-09-30T01:41:02.996 DEBUG:teuthology.parallel:result is None 2023-09-30T01:41:02.997 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-30T01:41:03.003 INFO:teuthology.task.internal:Creating test directory... 2023-09-30T01:41:03.003 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T01:41:03.006 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T01:41:03.024 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-30T01:41:03.030 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-30T01:41:03.039 INFO:teuthology.task.internal:Creating archive directory... 2023-09-30T01:41:03.039 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T01:41:03.063 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T01:41:03.100 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-30T01:41:03.105 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-30T01:41:03.106 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T01:41:03.127 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T01:41:03.156 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:41:03.168 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:41:03.179 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:41:03.192 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:41:03.193 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-30T01:41:03.200 INFO:teuthology.task.internal:Configuring sudo... 2023-09-30T01:41:03.200 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T01:41:03.213 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T01:41:03.263 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-30T01:41:03.271 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-30T01:41:03.271 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T01:41:03.281 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T01:41:03.320 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T01:41:03.376 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T01:41:03.421 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T01:41:03.501 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T01:41:03.546 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-09-30T01:41:03.546 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T01:41:03.612 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T01:41:03.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T01:41:03.704 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T01:41:03.788 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T01:41:03.831 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-09-30T01:41:03.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T01:41:03.899 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2023-09-30T01:41:03.901 DEBUG:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2023-09-30T01:41:03.950 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T01:41:04.127 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T01:41:04.282 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-30T01:41:04.291 INFO:teuthology.task.internal:Starting timer... 2023-09-30T01:41:04.292 INFO:teuthology.run_tasks:Running task pcp... 2023-09-30T01:41:04.302 INFO:teuthology.run_tasks:Running task selinux... 2023-09-30T01:41:04.310 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-09-30T01:41:04.311 DEBUG:teuthology.orchestra.run.smithi007:> sudo service auditd rotate 2023-09-30T01:41:04.377 INFO:teuthology.orchestra.run.smithi007.stdout:Rotating logs: [ OK ] 2023-09-30T01:41:04.378 DEBUG:teuthology.orchestra.run.smithi050:> sudo service auditd rotate 2023-09-30T01:41:04.452 INFO:teuthology.orchestra.run.smithi050.stdout:Rotating logs: [ OK ] 2023-09-30T01:41:04.455 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-30T01:41:04.455 DEBUG:teuthology.orchestra.run.smithi007:> /usr/sbin/getenforce 2023-09-30T01:41:04.480 INFO:teuthology.orchestra.run.smithi007.stdout:Permissive 2023-09-30T01:41:04.480 DEBUG:teuthology.orchestra.run.smithi050:> /usr/sbin/getenforce 2023-09-30T01:41:04.502 INFO:teuthology.orchestra.run.smithi050.stdout:Permissive 2023-09-30T01:41:04.503 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi007.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi050.front.sepia.ceph.com': 'permissive'} 2023-09-30T01:41:04.503 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T01:41:04.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:41:04.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T01:41:04.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:41:04.572 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-30T01:41:04.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo /usr/sbin/setenforce permissive 2023-09-30T01:41:04.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo /usr/sbin/setenforce permissive 2023-09-30T01:41:04.640 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-30T01:41:04.651 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-09-30T01:41:04.656 INFO:teuthology.repo_utils:Fetching main from origin 2023-09-30T01:41:04.796 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-30T01:41:04.811 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-30T01:41:04.812 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi050.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-30T01:49:30.009 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi050.front.sepia.ceph.com')] 2023-09-30T01:49:30.010 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-09-30T01:49:30.011 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:49:30.080 DEBUG:teuthology.orchestra.run.smithi007:> true 2023-09-30T01:49:30.150 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-09-30T01:49:30.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-09-30T01:49:30.151 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:49:30.220 DEBUG:teuthology.orchestra.run.smithi050:> true 2023-09-30T01:49:30.285 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2023-09-30T01:49:30.285 INFO:teuthology.run_tasks:Running task clock... 2023-09-30T01:49:30.296 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-30T01:49:30.296 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T01:49:30.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T01:49:30.299 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T01:49:30.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T01:49:30.328 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-30T01:49:30.343 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-30T01:49:30.357 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-30T01:49:30.373 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-30T01:49:30.382 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: ntpd: command not found 2023-09-30T01:49:30.394 INFO:teuthology.orchestra.run.smithi007.stdout:506 Cannot talk to daemon 2023-09-30T01:49:30.407 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: ntpd: command not found 2023-09-30T01:49:30.407 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-30T01:49:30.418 INFO:teuthology.orchestra.run.smithi050.stdout:506 Cannot talk to daemon 2023-09-30T01:49:30.421 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-30T01:49:30.432 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-30T01:49:30.447 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-30T01:49:30.497 INFO:teuthology.orchestra.run.smithi007.stderr:bash: ntpq: command not found 2023-09-30T01:49:30.499 INFO:teuthology.orchestra.run.smithi007.stdout:210 Number of sources = 4 2023-09-30T01:49:30.499 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T01:49:30.500 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2023-09-30T01:49:30.500 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:49:30.500 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:49:30.500 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:49:30.500 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:49:30.515 INFO:teuthology.orchestra.run.smithi050.stderr:bash: ntpq: command not found 2023-09-30T01:49:30.517 INFO:teuthology.orchestra.run.smithi050.stdout:210 Number of sources = 4 2023-09-30T01:49:30.517 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T01:49:30.517 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2023-09-30T01:49:30.518 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:49:30.518 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:49:30.518 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:49:30.518 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:49:30.518 INFO:teuthology.run_tasks:Running task install... 2023-09-30T01:49:30.526 DEBUG:teuthology.task.install:project ceph 2023-09-30T01:49:30.527 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}} 2023-09-30T01:49:30.527 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'} 2023-09-30T01:49:30.527 INFO:teuthology.task.install:Using flavor: default 2023-09-30T01:49:30.533 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-30T01:49:30.533 INFO:teuthology.task.install:extra packages: [] 2023-09-30T01:49:30.534 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-30T01:49:30.534 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:49:30.536 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-30T01:49:30.536 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:49:30.759 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-30T01:49:30.760 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-30T01:49:30.778 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-30T01:49:30.778 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-30T01:49:30.891 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-30T01:49:30.891 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-09-30T01:49:30.891 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-30T01:49:30.919 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-30T01:49:30.920 DEBUG:teuthology.orchestra.run.smithi050:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-30T01:49:30.925 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-30T01:49:30.925 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-09-30T01:49:30.925 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-30T01:49:30.952 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-30T01:49:30.953 DEBUG:teuthology.orchestra.run.smithi007:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-30T01:49:30.987 DEBUG:teuthology.orchestra.run.smithi050:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-30T01:49:31.019 DEBUG:teuthology.orchestra.run.smithi007:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-30T01:49:31.065 DEBUG:teuthology.orchestra.run.smithi050:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-30T01:49:31.095 DEBUG:teuthology.orchestra.run.smithi007:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-30T01:49:31.131 INFO:teuthology.orchestra.run.smithi050.stdout:check_obsoletes = 1 2023-09-30T01:49:31.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum clean all 2023-09-30T01:49:31.159 INFO:teuthology.orchestra.run.smithi007.stdout:check_obsoletes = 1 2023-09-30T01:49:31.161 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum clean all 2023-09-30T01:49:31.544 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:49:31.571 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:49:32.058 INFO:teuthology.orchestra.run.smithi007.stdout:44 files removed 2023-09-30T01:49:32.075 INFO:teuthology.orchestra.run.smithi050.stdout:44 files removed 2023-09-30T01:49:32.109 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-radosgw 2023-09-30T01:49:32.128 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-radosgw 2023-09-30T01:49:32.482 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:49:32.497 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:49:33.370 INFO:teuthology.orchestra.run.smithi007.stdout:ceph packages for x86_64 252 kB/s | 78 kB 00:00 2023-09-30T01:49:33.436 INFO:teuthology.orchestra.run.smithi050.stdout:ceph packages for x86_64 210 kB/s | 78 kB 00:00 2023-09-30T01:49:33.719 INFO:teuthology.orchestra.run.smithi007.stdout:ceph noarch packages 30 kB/s | 9.2 kB 00:00 2023-09-30T01:49:33.722 INFO:teuthology.orchestra.run.smithi050.stdout:ceph noarch packages 38 kB/s | 9.2 kB 00:00 2023-09-30T01:49:33.995 INFO:teuthology.orchestra.run.smithi007.stdout:ceph source packages 6.0 kB/s | 1.5 kB 00:00 2023-09-30T01:49:33.998 INFO:teuthology.orchestra.run.smithi050.stdout:ceph source packages 6.0 kB/s | 1.5 kB 00:00 2023-09-30T01:49:34.106 INFO:teuthology.orchestra.run.smithi007.stdout:Copr repo for python3-asyncssh owned by ceph 44 kB/s | 3.5 kB 00:00 2023-09-30T01:49:34.109 INFO:teuthology.orchestra.run.smithi050.stdout:Copr repo for python3-asyncssh owned by ceph 44 kB/s | 3.5 kB 00:00 2023-09-30T01:49:34.695 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 29 MB/s | 16 MB 00:00 2023-09-30T01:49:35.044 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 18 MB/s | 16 MB 00:00 2023-09-30T01:49:38.214 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2023-09-30T01:49:38.548 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2023-09-30T01:49:38.667 INFO:teuthology.orchestra.run.smithi050.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 104 MB/s | 38 MB 00:00 2023-09-30T01:49:38.983 INFO:teuthology.orchestra.run.smithi007.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 95 MB/s | 33 MB 00:00 2023-09-30T01:49:44.545 INFO:teuthology.orchestra.run.smithi007.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 104 MB/s | 38 MB 00:00 2023-09-30T01:49:44.628 INFO:teuthology.orchestra.run.smithi050.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 105 MB/s | 33 MB 00:00 2023-09-30T01:49:53.503 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:49:53.506 INFO:teuthology.orchestra.run.smithi050.stdout:==================================================================================================== 2023-09-30T01:49:53.506 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-09-30T01:49:53.506 INFO:teuthology.orchestra.run.smithi050.stdout:==================================================================================================== 2023-09-30T01:49:53.506 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-09-30T01:49:53.506 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T01:49:53.506 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrading: 2023-09-30T01:49:53.506 INFO:teuthology.orchestra.run.smithi050.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T01:49:53.507 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-30T01:49:53.507 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-09-30T01:49:53.507 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-30T01:49:53.507 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-30T01:49:53.507 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-30T01:49:53.507 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-30T01:49:53.507 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-30T01:49:53.507 INFO:teuthology.orchestra.run.smithi050.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-30T01:49:53.507 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-09-30T01:49:53.507 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-30T01:49:53.507 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-09-30T01:49:53.508 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-30T01:49:53.508 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-30T01:49:53.508 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-09-30T01:49:53.508 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-30T01:49:53.508 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-30T01:49:53.508 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-30T01:49:53.508 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-09-30T01:49:53.508 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-30T01:49:53.508 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-30T01:49:53.508 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-30T01:49:53.508 INFO:teuthology.orchestra.run.smithi050.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-30T01:49:53.509 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:49:53.509 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-09-30T01:49:53.509 INFO:teuthology.orchestra.run.smithi050.stdout:==================================================================================================== 2023-09-30T01:49:53.509 INFO:teuthology.orchestra.run.smithi050.stdout:Install 21 Packages 2023-09-30T01:49:53.509 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrade 2 Packages 2023-09-30T01:49:53.509 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:49:53.509 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 63 M 2023-09-30T01:49:53.510 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-09-30T01:49:54.204 INFO:teuthology.orchestra.run.smithi050.stdout:(1/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 7.8 MB/s | 5.2 MB 00:00 2023-09-30T01:49:54.248 INFO:teuthology.orchestra.run.smithi050.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 563 kB/s | 24 kB 00:00 2023-09-30T01:49:54.358 INFO:teuthology.orchestra.run.smithi050.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 7.1 MB/s | 796 kB 00:00 2023-09-30T01:49:54.419 INFO:teuthology.orchestra.run.smithi050.stdout:(4/23): libradosstriper1-17.2.6-1165.g93dce5e1. 8.1 MB/s | 506 kB 00:00 2023-09-30T01:49:54.883 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:49:54.886 INFO:teuthology.orchestra.run.smithi007.stdout:==================================================================================================== 2023-09-30T01:49:54.886 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-09-30T01:49:54.886 INFO:teuthology.orchestra.run.smithi007.stdout:==================================================================================================== 2023-09-30T01:49:54.886 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-09-30T01:49:54.886 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T01:49:54.886 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrading: 2023-09-30T01:49:54.886 INFO:teuthology.orchestra.run.smithi007.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T01:49:54.886 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-30T01:49:54.887 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-09-30T01:49:54.887 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-30T01:49:54.887 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-30T01:49:54.887 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-30T01:49:54.887 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-30T01:49:54.887 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-30T01:49:54.887 INFO:teuthology.orchestra.run.smithi007.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-30T01:49:54.887 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-09-30T01:49:54.887 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-30T01:49:54.887 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-09-30T01:49:54.887 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-30T01:49:54.887 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-30T01:49:54.888 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-09-30T01:49:54.888 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-30T01:49:54.888 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-30T01:49:54.888 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-30T01:49:54.888 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-09-30T01:49:54.888 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-30T01:49:54.888 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-30T01:49:54.888 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-30T01:49:54.888 INFO:teuthology.orchestra.run.smithi007.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-30T01:49:54.888 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:49:54.888 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-09-30T01:49:54.888 INFO:teuthology.orchestra.run.smithi007.stdout:==================================================================================================== 2023-09-30T01:49:54.889 INFO:teuthology.orchestra.run.smithi007.stdout:Install 21 Packages 2023-09-30T01:49:54.889 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrade 2 Packages 2023-09-30T01:49:54.889 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:49:54.889 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 63 M 2023-09-30T01:49:54.889 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-09-30T01:49:55.009 INFO:teuthology.orchestra.run.smithi050.stdout:(5/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_6 9.4 MB/s | 5.6 MB 00:00 2023-09-30T01:49:55.056 INFO:teuthology.orchestra.run.smithi050.stdout:(6/23): python3-ceph-argparse-17.2.6-1165.g93dc 951 kB/s | 45 kB 00:00 2023-09-30T01:49:55.105 INFO:teuthology.orchestra.run.smithi050.stdout:(7/23): python3-ceph-common-17.2.6-1165.g93dce5 2.3 MB/s | 115 kB 00:00 2023-09-30T01:49:55.158 INFO:teuthology.orchestra.run.smithi050.stdout:(8/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 3.5 MB/s | 190 kB 00:00 2023-09-30T01:49:55.220 INFO:teuthology.orchestra.run.smithi050.stdout:(9/23): python3-rados-17.2.6-1165.g93dce5e1.el8 5.9 MB/s | 370 kB 00:00 2023-09-30T01:49:55.285 INFO:teuthology.orchestra.run.smithi050.stdout:(10/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 5.4 MB/s | 358 kB 00:00 2023-09-30T01:49:55.344 INFO:teuthology.orchestra.run.smithi050.stdout:(11/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 1.8 MB/s | 107 kB 00:00 2023-09-30T01:49:55.473 INFO:teuthology.orchestra.run.smithi050.stdout:(12/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8 8.7 MB/s | 17 MB 00:01 2023-09-30T01:49:55.586 INFO:teuthology.orchestra.run.smithi050.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 522 kB/s | 59 kB 00:00 2023-09-30T01:49:55.620 INFO:teuthology.orchestra.run.smithi050.stdout:(14/23): libunwind-1.3.1-3.el8.x86_64.rpm 2.2 MB/s | 75 kB 00:00 2023-09-30T01:49:55.632 INFO:teuthology.orchestra.run.smithi050.stdout:(15/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.0 MB/s | 306 kB 00:00 2023-09-30T01:49:55.673 INFO:teuthology.orchestra.run.smithi050.stdout:(16/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2023-09-30T01:49:55.691 INFO:teuthology.orchestra.run.smithi050.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 2.5 MB/s | 44 kB 00:00 2023-09-30T01:49:55.710 INFO:teuthology.orchestra.run.smithi050.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 13 MB/s | 259 kB 00:00 2023-09-30T01:49:55.731 INFO:teuthology.orchestra.run.smithi050.stdout:(19/23): librdkafka-0.11.4-1.el8.x86_64.rpm 17 MB/s | 353 kB 00:00 2023-09-30T01:49:55.977 INFO:teuthology.orchestra.run.smithi050.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 4.9 MB/s | 1.7 MB 00:00 2023-09-30T01:49:56.007 INFO:teuthology.orchestra.run.smithi007.stdout:(1/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 5.3 MB/s | 5.2 MB 00:00 2023-09-30T01:49:56.046 INFO:teuthology.orchestra.run.smithi007.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 633 kB/s | 24 kB 00:00 2023-09-30T01:49:56.116 INFO:teuthology.orchestra.run.smithi050.stdout:(21/23): ceph-common-17.2.6-1165.g93dce5e1.el8. 9.1 MB/s | 23 MB 00:02 2023-09-30T01:49:56.157 INFO:teuthology.orchestra.run.smithi007.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 7.0 MB/s | 796 kB 00:00 2023-09-30T01:49:56.232 INFO:teuthology.orchestra.run.smithi007.stdout:(4/23): libradosstriper1-17.2.6-1165.g93dce5e1. 6.6 MB/s | 506 kB 00:00 2023-09-30T01:49:56.368 INFO:teuthology.orchestra.run.smithi050.stdout:(22/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 5.4 MB/s | 3.5 MB 00:00 2023-09-30T01:49:56.486 INFO:teuthology.orchestra.run.smithi050.stdout:(23/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 7.0 MB/s | 3.6 MB 00:00 2023-09-30T01:49:56.491 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:49:56.491 INFO:teuthology.orchestra.run.smithi050.stdout:Total 21 MB/s | 63 MB 00:02 2023-09-30T01:49:56.496 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-09-30T01:49:56.544 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-09-30T01:49:56.545 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-09-30T01:49:56.771 INFO:teuthology.orchestra.run.smithi007.stdout:(5/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_6 10 MB/s | 5.6 MB 00:00 2023-09-30T01:49:56.798 INFO:teuthology.orchestra.run.smithi007.stdout:(6/23): python3-ceph-argparse-17.2.6-1165.g93dc 1.7 MB/s | 45 kB 00:00 2023-09-30T01:49:56.823 INFO:teuthology.orchestra.run.smithi007.stdout:(7/23): python3-ceph-common-17.2.6-1165.g93dce5 4.5 MB/s | 115 kB 00:00 2023-09-30T01:49:56.854 INFO:teuthology.orchestra.run.smithi007.stdout:(8/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 6.1 MB/s | 190 kB 00:00 2023-09-30T01:49:56.891 INFO:teuthology.orchestra.run.smithi007.stdout:(9/23): python3-rados-17.2.6-1165.g93dce5e1.el8 9.8 MB/s | 370 kB 00:00 2023-09-30T01:49:56.930 INFO:teuthology.orchestra.run.smithi007.stdout:(10/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 9.0 MB/s | 358 kB 00:00 2023-09-30T01:49:56.960 INFO:teuthology.orchestra.run.smithi007.stdout:(11/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 3.5 MB/s | 107 kB 00:00 2023-09-30T01:49:57.026 INFO:teuthology.orchestra.run.smithi007.stdout:(12/23): gperftools-libs-2.7-9.el8.x86_64.rpm 4.5 MB/s | 306 kB 00:00 2023-09-30T01:49:57.035 INFO:teuthology.orchestra.run.smithi007.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 6.6 MB/s | 59 kB 00:00 2023-09-30T01:49:57.044 INFO:teuthology.orchestra.run.smithi007.stdout:(14/23): libunwind-1.3.1-3.el8.x86_64.rpm 8.2 MB/s | 75 kB 00:00 2023-09-30T01:49:57.078 INFO:teuthology.orchestra.run.smithi007.stdout:(15/23): thrift-0.13.0-2.el8.x86_64.rpm 52 MB/s | 1.7 MB 00:00 2023-09-30T01:49:57.119 INFO:teuthology.orchestra.run.smithi007.stdout:(16/23): python3-prettytable-0.7.2-14.el8.noarc 1.0 MB/s | 44 kB 00:00 2023-09-30T01:49:57.139 INFO:teuthology.orchestra.run.smithi007.stdout:(17/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 13 MB/s | 259 kB 00:00 2023-09-30T01:49:57.155 INFO:teuthology.orchestra.run.smithi007.stdout:(18/23): librdkafka-0.11.4-1.el8.x86_64.rpm 21 MB/s | 353 kB 00:00 2023-09-30T01:49:57.168 INFO:teuthology.orchestra.run.smithi007.stdout:(19/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 3.7 MB/s | 47 kB 00:00 2023-09-30T01:49:57.298 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-09-30T01:49:57.303 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-09-30T01:49:57.484 INFO:teuthology.orchestra.run.smithi007.stdout:(20/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 11 MB/s | 3.5 MB 00:00 2023-09-30T01:49:57.844 INFO:teuthology.orchestra.run.smithi007.stdout:(21/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 10 MB/s | 3.6 MB 00:00 2023-09-30T01:49:58.188 INFO:teuthology.orchestra.run.smithi007.stdout:(22/23): ceph-common-17.2.6-1165.g93dce5e1.el8. 7.4 MB/s | 23 MB 00:03 2023-09-30T01:49:58.524 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-09-30T01:49:58.701 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-09-30T01:49:59.134 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-09-30T01:49:59.368 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T01:49:59.476 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T01:49:59.700 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T01:49:59.983 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2023-09-30T01:50:00.193 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T01:50:00.620 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T01:50:00.859 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T01:50:00.956 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T01:50:01.151 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 7/25 2023-09-30T01:50:01.219 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 7/25 2023-09-30T01:50:01.574 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-09-30T01:50:01.930 INFO:teuthology.orchestra.run.smithi007.stdout:(23/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8 2.4 MB/s | 17 MB 00:06 2023-09-30T01:50:01.933 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:50:01.933 INFO:teuthology.orchestra.run.smithi007.stdout:Total 9.0 MB/s | 63 MB 00:07 2023-09-30T01:50:01.938 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-09-30T01:50:01.989 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-09-30T01:50:01.989 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-09-30T01:50:02.300 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/25 2023-09-30T01:50:02.492 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T01:50:02.549 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T01:50:02.680 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-30T01:50:02.711 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-09-30T01:50:02.716 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-09-30T01:50:02.897 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T01:50:03.165 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T01:50:03.442 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-30T01:50:03.634 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T01:50:03.705 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T01:50:03.871 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-30T01:50:04.047 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-09-30T01:50:04.078 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2023-09-30T01:50:04.250 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-09-30T01:50:04.356 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-30T01:50:04.559 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-30T01:50:04.672 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2023-09-30T01:50:04.875 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T01:50:05.009 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T01:50:05.199 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T01:50:05.467 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2023-09-30T01:50:05.675 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T01:50:06.085 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T01:50:06.284 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T01:50:06.352 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T01:50:06.569 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2023-09-30T01:50:06.775 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2023-09-30T01:50:07.010 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2023-09-30T01:50:07.720 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/25 2023-09-30T01:50:07.908 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T01:50:07.981 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T01:50:08.137 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-30T01:50:08.329 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T01:50:08.571 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T01:50:08.824 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-30T01:50:08.983 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T01:50:09.062 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T01:50:09.279 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-30T01:50:09.456 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2023-09-30T01:50:09.740 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-30T01:50:09.909 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T01:50:09.950 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-30T01:50:10.159 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T01:50:10.803 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T01:50:11.114 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T01:50:11.210 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T01:50:11.358 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T01:50:15.497 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T01:50:15.732 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T01:50:16.361 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T01:50:16.685 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T01:50:16.771 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T01:50:16.915 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T01:50:29.842 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T01:50:29.843 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /sys 2023-09-30T01:50:29.843 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /proc 2023-09-30T01:50:29.843 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /dev 2023-09-30T01:50:29.843 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /run 2023-09-30T01:50:29.843 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /mnt 2023-09-30T01:50:29.843 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /var/tmp 2023-09-30T01:50:29.843 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /home 2023-09-30T01:50:29.843 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /tmp 2023-09-30T01:50:29.843 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /dev 2023-09-30T01:50:29.843 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:50:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T01:50:31.751 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T01:50:31.751 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T01:50:31.966 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T01:50:31.966 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T01:50:32.802 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T01:50:32.802 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-30T01:50:32.802 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T01:50:32.802 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T01:50:32.802 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T01:50:32.802 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T01:50:32.802 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-30T01:50:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-30T01:50:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-30T01:50:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-30T01:50:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T01:50:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-30T01:50:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T01:50:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T01:50:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-09-30T01:50:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-09-30T01:50:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-30T01:50:32.803 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-09-30T01:50:32.804 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 18/25 2023-09-30T01:50:32.804 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 19/25 2023-09-30T01:50:32.804 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/25 2023-09-30T01:50:32.804 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 21/25 2023-09-30T01:50:32.804 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T01:50:32.804 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-30T01:50:32.804 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-30T01:50:34.124 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T01:50:34.125 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:50 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:50:35.359 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T01:50:35.360 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /sys 2023-09-30T01:50:35.360 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /proc 2023-09-30T01:50:35.360 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /dev 2023-09-30T01:50:35.360 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /run 2023-09-30T01:50:35.360 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /mnt 2023-09-30T01:50:35.360 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /var/tmp 2023-09-30T01:50:35.360 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /home 2023-09-30T01:50:35.360 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /tmp 2023-09-30T01:50:35.360 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /dev 2023-09-30T01:50:35.360 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:50:36.088 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-09-30T01:50:36.211 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:50:36.211 INFO:teuthology.orchestra.run.smithi050.stdout:Upgraded: 2023-09-30T01:50:36.211 INFO:teuthology.orchestra.run.smithi050.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:36.211 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:36.211 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-09-30T01:50:36.211 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:36.212 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:36.212 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:36.212 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:36.212 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-30T01:50:36.212 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:36.212 INFO:teuthology.orchestra.run.smithi050.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-30T01:50:36.212 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-09-30T01:50:36.212 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:36.212 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-09-30T01:50:36.212 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:36.212 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-30T01:50:36.212 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-30T01:50:36.213 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:36.213 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:36.213 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:36.213 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-30T01:50:36.213 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:36.213 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:36.213 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:36.213 INFO:teuthology.orchestra.run.smithi050.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-30T01:50:36.213 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:50:36.213 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:50:36.348 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-test 2023-09-30T01:50:36.736 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:50:37.190 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T01:50:37.270 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T01:50:37.270 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T01:50:37.497 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T01:50:37.497 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T01:50:37.570 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:53 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:50:38.388 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T01:50:38.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-30T01:50:38.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T01:50:38.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T01:50:38.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T01:50:38.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T01:50:38.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-30T01:50:38.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-30T01:50:38.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-30T01:50:38.389 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-30T01:50:38.390 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T01:50:38.390 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-30T01:50:38.390 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T01:50:38.390 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T01:50:38.390 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-09-30T01:50:38.390 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-09-30T01:50:38.390 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-30T01:50:38.390 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-09-30T01:50:38.390 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-30T01:50:38.390 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2023-09-30T01:50:38.390 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 20/25 2023-09-30T01:50:38.390 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 21/25 2023-09-30T01:50:38.391 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T01:50:38.391 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-30T01:50:38.391 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-30T01:50:39.554 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:50:39.555 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================================ 2023-09-30T01:50:39.555 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-09-30T01:50:39.555 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================================ 2023-09-30T01:50:39.555 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-09-30T01:50:39.555 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-30T01:50:39.555 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-09-30T01:50:39.555 INFO:teuthology.orchestra.run.smithi050.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-09-30T01:50:39.555 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-30T01:50:39.555 INFO:teuthology.orchestra.run.smithi050.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-09-30T01:50:39.556 INFO:teuthology.orchestra.run.smithi050.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-09-30T01:50:39.556 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-09-30T01:50:39.556 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:50:39.556 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-09-30T01:50:39.556 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================================ 2023-09-30T01:50:39.556 INFO:teuthology.orchestra.run.smithi050.stdout:Install 6 Packages 2023-09-30T01:50:39.556 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:50:39.557 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 53 M 2023-09-30T01:50:39.557 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 233 M 2023-09-30T01:50:39.557 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-09-30T01:50:39.561 INFO:teuthology.orchestra.run.smithi050.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 20 MB/s | 70 kB 00:00 2023-09-30T01:50:39.603 INFO:teuthology.orchestra.run.smithi050.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 3.7 MB/s | 161 kB 00:00 2023-09-30T01:50:39.622 INFO:teuthology.orchestra.run.smithi050.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 9.8 MB/s | 187 kB 00:00 2023-09-30T01:50:39.642 INFO:teuthology.orchestra.run.smithi050.stdout:(4/6): socat-1.7.3.3-2.el8.x86_64.rpm 15 MB/s | 302 kB 00:00 2023-09-30T01:50:39.714 INFO:teuthology.orchestra.run.smithi050.stdout:(5/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 1.1 MB/s | 175 kB 00:00 2023-09-30T01:50:39.854 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T01:50:39.855 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:55 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:50:41.819 INFO:teuthology.orchestra.run.smithi007.stdout:Installed products updated. 2023-09-30T01:50:41.939 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:50:41.939 INFO:teuthology.orchestra.run.smithi007.stdout:Upgraded: 2023-09-30T01:50:41.939 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:41.939 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:41.939 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-09-30T01:50:41.939 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-30T01:50:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-30T01:50:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-09-30T01:50:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-09-30T01:50:41.940 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-30T01:50:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-30T01:50:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-30T01:50:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-30T01:50:41.941 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:50:41.941 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:50:42.090 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-test 2023-09-30T01:50:42.126 INFO:teuthology.orchestra.run.smithi050.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 20 MB/s | 52 MB 00:02 2023-09-30T01:50:42.129 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:50:42.130 INFO:teuthology.orchestra.run.smithi050.stdout:Total 20 MB/s | 53 MB 00:02 2023-09-30T01:50:42.134 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-09-30T01:50:42.161 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-09-30T01:50:42.161 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-09-30T01:50:42.477 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:50:42.523 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-09-30T01:50:42.528 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-09-30T01:50:43.077 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-09-30T01:50:43.315 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2023-09-30T01:50:43.426 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:59 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:50:43.520 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2023-09-30T01:50:43.601 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2023-09-30T01:50:43.773 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2023-09-30T01:50:43.971 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-09-30T01:50:44.153 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T01:50:45.365 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:50:45.366 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2023-09-30T01:50:45.366 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-09-30T01:50:45.366 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2023-09-30T01:50:45.366 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-09-30T01:50:45.366 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-30T01:50:45.367 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-09-30T01:50:45.367 INFO:teuthology.orchestra.run.smithi007.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-09-30T01:50:45.367 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-30T01:50:45.367 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-09-30T01:50:45.367 INFO:teuthology.orchestra.run.smithi007.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-09-30T01:50:45.367 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-09-30T01:50:45.367 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:50:45.367 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-09-30T01:50:45.367 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2023-09-30T01:50:45.367 INFO:teuthology.orchestra.run.smithi007.stdout:Install 6 Packages 2023-09-30T01:50:45.367 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:50:45.368 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 53 M 2023-09-30T01:50:45.368 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 233 M 2023-09-30T01:50:45.368 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-09-30T01:50:45.372 INFO:teuthology.orchestra.run.smithi007.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 19 MB/s | 70 kB 00:00 2023-09-30T01:50:45.410 INFO:teuthology.orchestra.run.smithi007.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 4.2 MB/s | 161 kB 00:00 2023-09-30T01:50:45.429 INFO:teuthology.orchestra.run.smithi007.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 9.6 MB/s | 187 kB 00:00 2023-09-30T01:50:45.450 INFO:teuthology.orchestra.run.smithi007.stdout:(4/6): socat-1.7.3.3-2.el8.x86_64.rpm 14 MB/s | 302 kB 00:00 2023-09-30T01:50:45.525 INFO:teuthology.orchestra.run.smithi007.stdout:(5/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 1.1 MB/s | 175 kB 00:00 2023-09-30T01:50:47.895 INFO:teuthology.orchestra.run.smithi007.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 20 MB/s | 52 MB 00:02 2023-09-30T01:50:47.898 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:50:47.900 INFO:teuthology.orchestra.run.smithi007.stdout:Total 21 MB/s | 53 MB 00:02 2023-09-30T01:50:47.904 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-09-30T01:50:47.930 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-09-30T01:50:47.931 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-09-30T01:50:48.319 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-09-30T01:50:48.323 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-09-30T01:50:48.877 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-09-30T01:50:49.110 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2023-09-30T01:50:49.285 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2023-09-30T01:50:49.410 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2023-09-30T01:50:49.459 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T01:50:49.567 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2023-09-30T01:50:49.743 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-09-30T01:50:49.777 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T01:50:49.876 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T01:50:50.053 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T01:50:50.054 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-30T01:50:50.054 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-30T01:50:50.054 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-09-30T01:50:50.054 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2023-09-30T01:50:50.054 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-09-30T01:50:51.118 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2023-09-30T01:50:51.118 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:07 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:50:53.189 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-09-30T01:50:53.299 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:50:53.299 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-09-30T01:50:53.299 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:53.299 INFO:teuthology.orchestra.run.smithi050.stdout: jq-1.5-12.el8.x86_64 2023-09-30T01:50:53.299 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:53.299 INFO:teuthology.orchestra.run.smithi050.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-30T01:50:53.299 INFO:teuthology.orchestra.run.smithi050.stdout: socat-1.7.3.3-2.el8.x86_64 2023-09-30T01:50:53.299 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-30T01:50:53.300 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:50:53.300 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:50:53.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph 2023-09-30T01:50:53.821 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:50:54.651 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:10 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:50:55.190 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T01:50:55.483 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T01:50:55.799 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T01:50:55.799 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-30T01:50:55.799 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-30T01:50:55.800 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-09-30T01:50:55.800 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2023-09-30T01:50:55.800 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-09-30T01:50:56.569 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:50:56.572 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================================= 2023-09-30T01:50:56.572 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-09-30T01:50:56.572 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================================= 2023-09-30T01:50:56.573 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-09-30T01:50:56.573 INFO:teuthology.orchestra.run.smithi050.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-30T01:50:56.573 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-09-30T01:50:56.573 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-30T01:50:56.573 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-30T01:50:56.573 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-30T01:50:56.573 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-30T01:50:56.573 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T01:50:56.573 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-30T01:50:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-30T01:50:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-30T01:50:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-30T01:50:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-30T01:50:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-30T01:50:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-30T01:50:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-09-30T01:50:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-09-30T01:50:56.574 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-30T01:50:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-30T01:50:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-30T01:50:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-09-30T01:50:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-30T01:50:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-30T01:50:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-30T01:50:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-30T01:50:56.575 INFO:teuthology.orchestra.run.smithi050.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-30T01:50:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-30T01:50:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-30T01:50:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-09-30T01:50:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-30T01:50:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:50:56.576 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-09-30T01:50:56.576 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================================= 2023-09-30T01:50:56.576 INFO:teuthology.orchestra.run.smithi050.stdout:Install 28 Packages 2023-09-30T01:50:56.576 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:50:56.577 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 27 M 2023-09-30T01:50:56.577 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 106 M 2023-09-30T01:50:56.577 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-09-30T01:50:56.877 INFO:teuthology.orchestra.run.smithi050.stdout:(1/28): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 43 kB/s | 6.4 kB 00:00 2023-09-30T01:50:56.941 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2023-09-30T01:50:56.942 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:12 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:50:57.246 INFO:teuthology.orchestra.run.smithi050.stdout:(2/28): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 3.0 MB/s | 1.6 MB 00:00 2023-09-30T01:50:57.320 INFO:teuthology.orchestra.run.smithi050.stdout:(3/28): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 3.9 MB/s | 2.3 MB 00:00 2023-09-30T01:50:57.366 INFO:teuthology.orchestra.run.smithi050.stdout:(4/28): ceph-mgr-modules-core-17.2.6-1165.g93dc 5.1 MB/s | 238 kB 00:00 2023-09-30T01:50:57.574 INFO:teuthology.orchestra.run.smithi050.stdout:(5/28): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86_ 5.6 MB/s | 3.9 MB 00:00 2023-09-30T01:50:57.672 INFO:teuthology.orchestra.run.smithi050.stdout:(6/28): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 142 kB/s | 44 kB 00:00 2023-09-30T01:50:57.889 INFO:teuthology.orchestra.run.smithi050.stdout:(7/28): python3-cheroot-8.5.2-1.el8.noarch.rpm 798 kB/s | 173 kB 00:00 2023-09-30T01:50:57.927 INFO:teuthology.orchestra.run.smithi050.stdout:(8/28): python3-beautifulsoup4-4.6.3-2.el8.1.no 524 kB/s | 185 kB 00:00 2023-09-30T01:50:57.998 INFO:teuthology.orchestra.run.smithi050.stdout:(9/28): python3-jaraco-6.2-6.el8.noarch.rpm 157 kB/s | 11 kB 00:00 2023-09-30T01:50:58.035 INFO:teuthology.orchestra.run.smithi050.stdout:(10/28): python3-cherrypy-18.4.0-1.el8.noarch.r 2.6 MB/s | 384 kB 00:00 2023-09-30T01:50:58.069 INFO:teuthology.orchestra.run.smithi050.stdout:(11/28): python3-jaraco-functools-2.0-4.el8.noa 250 kB/s | 18 kB 00:00 2023-09-30T01:50:58.109 INFO:teuthology.orchestra.run.smithi050.stdout:(12/28): python3-logutils-0.3.5-11.el8.noarch.r 667 kB/s | 49 kB 00:00 2023-09-30T01:50:58.140 INFO:teuthology.orchestra.run.smithi050.stdout:(13/28): python3-more-itertools-7.2.0-3.el8.noa 831 kB/s | 59 kB 00:00 2023-09-30T01:50:58.185 INFO:teuthology.orchestra.run.smithi050.stdout:(14/28): python3-pecan-1.3.2-9.el8.noarch.rpm 3.6 MB/s | 283 kB 00:00 2023-09-30T01:50:58.210 INFO:teuthology.orchestra.run.smithi050.stdout:(15/28): python3-portend-2.6-1.el8.noarch.rpm 222 kB/s | 16 kB 00:00 2023-09-30T01:50:58.258 INFO:teuthology.orchestra.run.smithi050.stdout:(16/28): python3-simplegeneric-0.8.1-17.el8.noa 257 kB/s | 19 kB 00:00 2023-09-30T01:50:58.280 INFO:teuthology.orchestra.run.smithi050.stdout:(17/28): python3-singledispatch-3.4.0.3-18.el8. 340 kB/s | 24 kB 00:00 2023-09-30T01:50:58.331 INFO:teuthology.orchestra.run.smithi050.stdout:(18/28): python3-tempora-1.14.1-5.el8.noarch.rp 401 kB/s | 29 kB 00:00 2023-09-30T01:50:58.351 INFO:teuthology.orchestra.run.smithi050.stdout:(19/28): python3-trustme-0.6.0-4.el8.noarch.rpm 390 kB/s | 27 kB 00:00 2023-09-30T01:50:58.407 INFO:teuthology.orchestra.run.smithi050.stdout:(20/28): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2023-09-30T01:50:58.481 INFO:teuthology.orchestra.run.smithi050.stdout:(21/28): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2023-09-30T01:50:58.492 INFO:teuthology.orchestra.run.smithi050.stdout:(22/28): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.7 MB/s | 251 kB 00:00 2023-09-30T01:50:58.554 INFO:teuthology.orchestra.run.smithi050.stdout:(23/28): python3-zc-lockfile-2.0-2.el8.noarch.r 315 kB/s | 23 kB 00:00 2023-09-30T01:50:58.631 INFO:teuthology.orchestra.run.smithi050.stdout:(24/28): python3-werkzeug-0.12.2-4.el8.noarch.r 3.2 MB/s | 457 kB 00:00 2023-09-30T01:50:58.642 INFO:teuthology.orchestra.run.smithi050.stdout:(25/28): python3-markupsafe-0.23-19.el8.x86_64. 440 kB/s | 39 kB 00:00 2023-09-30T01:50:58.711 INFO:teuthology.orchestra.run.smithi050.stdout:(26/28): python3-mako-1.0.6-13.el8.noarch.rpm 1.9 MB/s | 157 kB 00:00 2023-09-30T01:50:58.758 INFO:teuthology.orchestra.run.smithi050.stdout:(27/28): python3-pytz-2017.2-9.el8.noarch.rpm 469 kB/s | 54 kB 00:00 2023-09-30T01:50:58.862 INFO:teuthology.orchestra.run.smithi050.stdout:(28/28): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86 10 MB/s | 17 MB 00:01 2023-09-30T01:50:58.865 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:50:58.865 INFO:teuthology.orchestra.run.smithi050.stdout:Total 12 MB/s | 27 MB 00:02 2023-09-30T01:50:58.870 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-09-30T01:50:58.911 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-09-30T01:50:58.911 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-09-30T01:50:59.315 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-09-30T01:50:59.319 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-09-30T01:51:00.449 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-09-30T01:51:00.572 INFO:teuthology.orchestra.run.smithi007.stdout:Installed products updated. 2023-09-30T01:51:00.678 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:51:00.679 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-09-30T01:51:00.679 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:00.679 INFO:teuthology.orchestra.run.smithi007.stdout: jq-1.5-12.el8.x86_64 2023-09-30T01:51:00.679 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:00.679 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-30T01:51:00.679 INFO:teuthology.orchestra.run.smithi007.stdout: socat-1.7.3.3-2.el8.x86_64 2023-09-30T01:51:00.679 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-30T01:51:00.679 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:51:00.679 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:51:00.686 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/28 2023-09-30T01:51:00.805 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph 2023-09-30T01:51:00.901 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/28 2023-09-30T01:51:01.107 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 3/28 2023-09-30T01:51:01.179 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:51:01.379 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/28 2023-09-30T01:51:01.718 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/28 2023-09-30T01:51:01.978 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/28 2023-09-30T01:51:02.231 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 7/28 2023-09-30T01:51:02.460 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 8/28 2023-09-30T01:51:02.506 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:18 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:51:02.609 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 9/28 2023-09-30T01:51:02.790 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 10/28 2023-09-30T01:51:03.026 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 11/28 2023-09-30T01:51:03.255 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 12/28 2023-09-30T01:51:03.409 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 13/28 2023-09-30T01:51:03.668 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 14/28 2023-09-30T01:51:03.891 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 15/28 2023-09-30T01:51:04.050 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 16/28 2023-09-30T01:51:04.424 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:51:04.427 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================================= 2023-09-30T01:51:04.427 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-09-30T01:51:04.427 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================================= 2023-09-30T01:51:04.427 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-09-30T01:51:04.427 INFO:teuthology.orchestra.run.smithi007.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-30T01:51:04.427 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-09-30T01:51:04.427 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-30T01:51:04.427 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-30T01:51:04.428 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-30T01:51:04.428 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-30T01:51:04.428 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T01:51:04.428 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-30T01:51:04.428 INFO:teuthology.orchestra.run.smithi007.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-30T01:51:04.428 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-30T01:51:04.428 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-30T01:51:04.428 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-30T01:51:04.428 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-30T01:51:04.428 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-30T01:51:04.429 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-09-30T01:51:04.429 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-09-30T01:51:04.429 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-30T01:51:04.429 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-30T01:51:04.429 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-30T01:51:04.429 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-09-30T01:51:04.429 INFO:teuthology.orchestra.run.smithi007.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-30T01:51:04.429 INFO:teuthology.orchestra.run.smithi007.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-30T01:51:04.429 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-30T01:51:04.429 INFO:teuthology.orchestra.run.smithi007.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-30T01:51:04.430 INFO:teuthology.orchestra.run.smithi007.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-30T01:51:04.430 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-30T01:51:04.430 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-30T01:51:04.430 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-09-30T01:51:04.430 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-30T01:51:04.430 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:51:04.430 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-09-30T01:51:04.430 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================================= 2023-09-30T01:51:04.430 INFO:teuthology.orchestra.run.smithi007.stdout:Install 28 Packages 2023-09-30T01:51:04.430 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:51:04.431 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 27 M 2023-09-30T01:51:04.431 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 106 M 2023-09-30T01:51:04.431 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-09-30T01:51:04.613 INFO:teuthology.orchestra.run.smithi007.stdout:(1/28): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 43 kB/s | 6.4 kB 00:00 2023-09-30T01:51:04.924 INFO:teuthology.orchestra.run.smithi007.stdout:(2/28): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 3.4 MB/s | 1.6 MB 00:00 2023-09-30T01:51:05.082 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-portend-2.6-1.el8.noarch 17/28 2023-09-30T01:51:05.303 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 18/28 2023-09-30T01:51:05.449 INFO:teuthology.orchestra.run.smithi007.stdout:(3/28): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 2.3 MB/s | 2.3 MB 00:00 2023-09-30T01:51:05.526 INFO:teuthology.orchestra.run.smithi007.stdout:(4/28): ceph-mgr-modules-core-17.2.6-1165.g93dc 3.0 MB/s | 238 kB 00:00 2023-09-30T01:51:05.540 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/28 2023-09-30T01:51:05.825 INFO:teuthology.orchestra.run.smithi007.stdout:(5/28): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86_ 3.2 MB/s | 3.9 MB 00:01 2023-09-30T01:51:05.883 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 20/28 2023-09-30T01:51:05.938 INFO:teuthology.orchestra.run.smithi007.stdout:(6/28): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86_ 16 MB/s | 17 MB 00:01 2023-09-30T01:51:06.018 INFO:teuthology.orchestra.run.smithi007.stdout:(7/28): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 88 kB/s | 44 kB 00:00 2023-09-30T01:51:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 21/28 2023-09-30T01:51:06.339 INFO:teuthology.orchestra.run.smithi007.stdout:(8/28): python3-cherrypy-18.4.0-1.el8.noarch.rp 1.2 MB/s | 384 kB 00:00 2023-09-30T01:51:06.420 INFO:teuthology.orchestra.run.smithi007.stdout:(9/28): python3-jaraco-6.2-6.el8.noarch.rpm 138 kB/s | 11 kB 00:00 2023-09-30T01:51:06.430 INFO:teuthology.orchestra.run.smithi007.stdout:(10/28): python3-beautifulsoup4-4.6.3-2.el8.1.n 306 kB/s | 185 kB 00:00 2023-09-30T01:51:06.462 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 22/28 2023-09-30T01:51:06.501 INFO:teuthology.orchestra.run.smithi007.stdout:(11/28): python3-jaraco-functools-2.0-4.el8.noa 218 kB/s | 18 kB 00:00 2023-09-30T01:51:06.511 INFO:teuthology.orchestra.run.smithi007.stdout:(12/28): python3-logutils-0.3.5-11.el8.noarch.r 605 kB/s | 49 kB 00:00 2023-09-30T01:51:06.513 INFO:teuthology.orchestra.run.smithi007.stdout:(13/28): python3-cheroot-8.5.2-1.el8.noarch.rpm 303 kB/s | 173 kB 00:00 2023-09-30T01:51:06.582 INFO:teuthology.orchestra.run.smithi007.stdout:(14/28): python3-more-itertools-7.2.0-3.el8.noa 726 kB/s | 59 kB 00:00 2023-09-30T01:51:06.594 INFO:teuthology.orchestra.run.smithi007.stdout:(15/28): python3-portend-2.6-1.el8.noarch.rpm 194 kB/s | 16 kB 00:00 2023-09-30T01:51:06.663 INFO:teuthology.orchestra.run.smithi007.stdout:(16/28): python3-simplegeneric-0.8.1-17.el8.noa 232 kB/s | 19 kB 00:00 2023-09-30T01:51:06.672 INFO:teuthology.orchestra.run.smithi007.stdout:(17/28): python3-pecan-1.3.2-9.el8.noarch.rpm 1.7 MB/s | 283 kB 00:00 2023-09-30T01:51:06.674 INFO:teuthology.orchestra.run.smithi007.stdout:(18/28): python3-singledispatch-3.4.0.3-18.el8. 297 kB/s | 24 kB 00:00 2023-09-30T01:51:06.743 INFO:teuthology.orchestra.run.smithi007.stdout:(19/28): python3-tempora-1.14.1-5.el8.noarch.rp 363 kB/s | 29 kB 00:00 2023-09-30T01:51:06.753 INFO:teuthology.orchestra.run.smithi007.stdout:(20/28): python3-trustme-0.6.0-4.el8.noarch.rpm 342 kB/s | 27 kB 00:00 2023-09-30T01:51:06.805 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 23/28 2023-09-30T01:51:06.826 INFO:teuthology.orchestra.run.smithi007.stdout:(21/28): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.0 MB/s | 251 kB 00:00 2023-09-30T01:51:06.834 INFO:teuthology.orchestra.run.smithi007.stdout:(22/28): python3-webtest-2.0.33-1.el8.noarch.rp 1.0 MB/s | 85 kB 00:00 2023-09-30T01:51:06.835 INFO:teuthology.orchestra.run.smithi007.stdout:(23/28): python3-waitress-1.4.3-1.el8.noarch.rp 1.5 MB/s | 255 kB 00:00 2023-09-30T01:51:06.887 INFO:teuthology.orchestra.run.smithi007.stdout:(24/28): python3-werkzeug-0.12.2-4.el8.noarch.r 8.5 MB/s | 457 kB 00:00 2023-09-30T01:51:06.892 INFO:teuthology.orchestra.run.smithi007.stdout:(25/28): python3-markupsafe-0.23-19.el8.x86_64. 685 kB/s | 39 kB 00:00 2023-09-30T01:51:06.907 INFO:teuthology.orchestra.run.smithi007.stdout:(26/28): python3-zc-lockfile-2.0-2.el8.noarch.r 285 kB/s | 23 kB 00:00 2023-09-30T01:51:06.910 INFO:teuthology.orchestra.run.smithi007.stdout:(27/28): python3-mako-1.0.6-13.el8.noarch.rpm 6.8 MB/s | 157 kB 00:00 2023-09-30T01:51:06.920 INFO:teuthology.orchestra.run.smithi007.stdout:(28/28): python3-pytz-2017.2-9.el8.noarch.rpm 2.0 MB/s | 54 kB 00:00 2023-09-30T01:51:06.926 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:51:06.927 INFO:teuthology.orchestra.run.smithi007.stdout:Total 11 MB/s | 27 MB 00:02 2023-09-30T01:51:06.932 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-09-30T01:51:06.965 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/28 2023-09-30T01:51:06.974 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-09-30T01:51:06.974 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-09-30T01:51:07.504 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-09-30T01:51:07.508 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-09-30T01:51:08.542 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/28 2023-09-30T01:51:08.618 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-09-30T01:51:08.732 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 25/28 2023-09-30T01:51:08.889 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/28 2023-09-30T01:51:09.150 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 25/28 2023-09-30T01:51:09.248 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/28 2023-09-30T01:51:09.334 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 26/28 2023-09-30T01:51:09.446 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 3/28 2023-09-30T01:51:09.565 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 26/28 2023-09-30T01:51:09.701 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 27/28 2023-09-30T01:51:09.720 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 27/28 2023-09-30T01:51:09.734 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/28 2023-09-30T01:51:09.867 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/28 2023-09-30T01:51:10.280 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/28 2023-09-30T01:51:10.281 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/28 2023-09-30T01:51:10.281 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/28 2023-09-30T01:51:10.281 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/28 2023-09-30T01:51:10.281 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/28 2023-09-30T01:51:10.281 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/28 2023-09-30T01:51:10.281 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/28 2023-09-30T01:51:10.281 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/28 2023-09-30T01:51:10.281 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/28 2023-09-30T01:51:10.281 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/28 2023-09-30T01:51:10.281 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/28 2023-09-30T01:51:10.281 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/28 2023-09-30T01:51:10.282 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/28 2023-09-30T01:51:10.282 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/28 2023-09-30T01:51:10.282 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/28 2023-09-30T01:51:10.282 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/28 2023-09-30T01:51:10.282 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/28 2023-09-30T01:51:10.282 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/28 2023-09-30T01:51:10.282 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/28 2023-09-30T01:51:10.282 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/28 2023-09-30T01:51:10.282 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/28 2023-09-30T01:51:10.282 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/28 2023-09-30T01:51:10.282 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/28 2023-09-30T01:51:10.283 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/28 2023-09-30T01:51:10.283 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/28 2023-09-30T01:51:10.283 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/28 2023-09-30T01:51:10.283 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/28 2023-09-30T01:51:10.283 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 27/28 2023-09-30T01:51:10.867 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/28 2023-09-30T01:51:11.134 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 6/28 2023-09-30T01:51:11.402 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 7/28 2023-09-30T01:51:11.708 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 8/28 2023-09-30T01:51:11.718 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/28 2023-09-30T01:51:11.718 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:27 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:51:11.899 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 9/28 2023-09-30T01:51:12.054 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 10/28 2023-09-30T01:51:12.274 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 11/28 2023-09-30T01:51:12.493 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 12/28 2023-09-30T01:51:12.647 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 13/28 2023-09-30T01:51:12.898 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 14/28 2023-09-30T01:51:13.129 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 15/28 2023-09-30T01:51:13.305 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 16/28 2023-09-30T01:51:13.605 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-portend-2.6-1.el8.noarch 17/28 2023-09-30T01:51:13.661 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-09-30T01:51:13.770 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:51:13.770 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-09-30T01:51:13.770 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:13.771 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:13.771 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:13.771 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:51:13.771 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:13.771 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:13.771 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-30T01:51:13.771 INFO:teuthology.orchestra.run.smithi050.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-30T01:51:13.771 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-30T01:51:13.771 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-30T01:51:13.772 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-30T01:51:13.772 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-30T01:51:13.772 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-30T01:51:13.772 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako-1.0.6-13.el8.noarch 2023-09-30T01:51:13.772 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-30T01:51:13.772 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-30T01:51:13.772 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-30T01:51:13.772 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend-2.6-1.el8.noarch 2023-09-30T01:51:13.772 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz-2017.2-9.el8.noarch 2023-09-30T01:51:13.772 INFO:teuthology.orchestra.run.smithi050.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-30T01:51:13.772 INFO:teuthology.orchestra.run.smithi050.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-30T01:51:13.773 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-30T01:51:13.773 INFO:teuthology.orchestra.run.smithi050.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-30T01:51:13.773 INFO:teuthology.orchestra.run.smithi050.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-30T01:51:13.773 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-30T01:51:13.773 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-30T01:51:13.773 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-30T01:51:13.773 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-30T01:51:13.773 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:51:13.773 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:51:13.850 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 18/28 2023-09-30T01:51:13.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-base 2023-09-30T01:51:14.095 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/28 2023-09-30T01:51:14.303 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:51:14.436 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 20/28 2023-09-30T01:51:14.689 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 21/28 2023-09-30T01:51:15.031 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 22/28 2023-09-30T01:51:15.164 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:31 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:51:15.392 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 23/28 2023-09-30T01:51:15.567 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/28 2023-09-30T01:51:16.732 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:51:17.165 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/28 2023-09-30T01:51:17.228 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:51:17.229 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-09-30T01:51:17.229 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:51:17.339 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install cephadm 2023-09-30T01:51:17.366 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 25/28 2023-09-30T01:51:17.723 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:51:17.786 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 25/28 2023-09-30T01:51:17.981 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 26/28 2023-09-30T01:51:18.221 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 26/28 2023-09-30T01:51:18.380 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 27/28 2023-09-30T01:51:18.400 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 27/28 2023-09-30T01:51:18.540 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/28 2023-09-30T01:51:18.668 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:34 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:51:18.958 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/28 2023-09-30T01:51:18.959 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/28 2023-09-30T01:51:18.959 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/28 2023-09-30T01:51:18.959 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/28 2023-09-30T01:51:18.959 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/28 2023-09-30T01:51:18.959 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/28 2023-09-30T01:51:18.960 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/28 2023-09-30T01:51:18.960 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/28 2023-09-30T01:51:18.960 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/28 2023-09-30T01:51:18.960 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/28 2023-09-30T01:51:18.960 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/28 2023-09-30T01:51:18.960 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/28 2023-09-30T01:51:18.960 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/28 2023-09-30T01:51:18.961 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/28 2023-09-30T01:51:18.961 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/28 2023-09-30T01:51:18.961 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/28 2023-09-30T01:51:18.961 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/28 2023-09-30T01:51:18.961 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/28 2023-09-30T01:51:18.961 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/28 2023-09-30T01:51:18.961 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/28 2023-09-30T01:51:18.961 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/28 2023-09-30T01:51:18.962 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/28 2023-09-30T01:51:18.962 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/28 2023-09-30T01:51:18.962 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/28 2023-09-30T01:51:18.962 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/28 2023-09-30T01:51:18.962 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 25/28 2023-09-30T01:51:18.962 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 26/28 2023-09-30T01:51:18.962 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 27/28 2023-09-30T01:51:20.570 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/28 2023-09-30T01:51:20.570 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:36 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:51:20.643 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:51:20.643 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:51:20.643 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-09-30T01:51:20.643 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:51:20.643 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-09-30T01:51:20.643 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-30T01:51:20.644 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:51:20.644 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-09-30T01:51:20.644 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:51:20.644 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-09-30T01:51:20.644 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:51:20.644 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 80 k 2023-09-30T01:51:20.645 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 363 k 2023-09-30T01:51:20.645 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-09-30T01:51:20.867 INFO:teuthology.orchestra.run.smithi050.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 360 kB/s | 80 kB 00:00 2023-09-30T01:51:20.867 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:51:20.868 INFO:teuthology.orchestra.run.smithi050.stdout:Total 357 kB/s | 80 kB 00:00 2023-09-30T01:51:20.868 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-09-30T01:51:20.877 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-09-30T01:51:20.877 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-09-30T01:51:20.879 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-09-30T01:51:20.884 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-09-30T01:51:20.937 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-09-30T01:51:22.560 INFO:teuthology.orchestra.run.smithi007.stdout:Installed products updated. 2023-09-30T01:51:22.669 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:51:22.669 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-09-30T01:51:22.669 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:22.669 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:22.670 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:22.670 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:51:22.670 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:22.670 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:22.670 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-30T01:51:22.670 INFO:teuthology.orchestra.run.smithi007.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-30T01:51:22.670 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-30T01:51:22.670 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-30T01:51:22.670 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-30T01:51:22.670 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-30T01:51:22.670 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-30T01:51:22.671 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako-1.0.6-13.el8.noarch 2023-09-30T01:51:22.671 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-30T01:51:22.671 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-30T01:51:22.671 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-30T01:51:22.671 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend-2.6-1.el8.noarch 2023-09-30T01:51:22.671 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz-2017.2-9.el8.noarch 2023-09-30T01:51:22.671 INFO:teuthology.orchestra.run.smithi007.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-30T01:51:22.671 INFO:teuthology.orchestra.run.smithi007.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-30T01:51:22.671 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-30T01:51:22.671 INFO:teuthology.orchestra.run.smithi007.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-30T01:51:22.671 INFO:teuthology.orchestra.run.smithi007.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-30T01:51:22.671 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-30T01:51:22.672 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-30T01:51:22.672 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-30T01:51:22.672 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-30T01:51:22.672 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:51:22.672 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:51:22.853 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-base 2023-09-30T01:51:23.238 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:51:24.159 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:40 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:51:25.153 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:51:25.301 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:51:25.524 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:51:25.652 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:51:26.170 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:51:26.171 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-09-30T01:51:26.171 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:51:26.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install cephadm 2023-09-30T01:51:26.491 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:51:26.492 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:42 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:51:26.660 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:51:27.608 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:43 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:51:28.468 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-09-30T01:51:28.574 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:51:28.574 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-09-30T01:51:28.574 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:51:28.575 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:51:28.575 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:51:28.711 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-immutable-object-cache 2023-09-30T01:51:29.108 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:51:29.538 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:51:29.539 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:51:29.539 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-09-30T01:51:29.539 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:51:29.539 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-09-30T01:51:29.539 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-30T01:51:29.539 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:51:29.539 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-09-30T01:51:29.539 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:51:29.539 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-09-30T01:51:29.539 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:51:29.540 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 80 k 2023-09-30T01:51:29.540 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 363 k 2023-09-30T01:51:29.540 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-09-30T01:51:29.765 INFO:teuthology.orchestra.run.smithi007.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 360 kB/s | 80 kB 00:00 2023-09-30T01:51:29.766 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:51:29.766 INFO:teuthology.orchestra.run.smithi007.stdout:Total 358 kB/s | 80 kB 00:00 2023-09-30T01:51:29.766 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-09-30T01:51:29.772 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-09-30T01:51:29.773 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-09-30T01:51:29.774 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-09-30T01:51:29.780 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-09-30T01:51:29.823 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-09-30T01:51:30.024 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:46 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:51:32.013 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:51:32.014 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:51:32.014 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2023-09-30T01:51:32.014 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:51:32.014 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-09-30T01:51:32.015 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-30T01:51:32.015 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:51:32.015 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-09-30T01:51:32.015 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:51:32.015 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-09-30T01:51:32.015 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:51:32.015 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 161 k 2023-09-30T01:51:32.015 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 490 k 2023-09-30T01:51:32.015 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-09-30T01:51:32.274 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 625 kB/s | 161 kB 00:00 2023-09-30T01:51:32.274 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:51:32.275 INFO:teuthology.orchestra.run.smithi050.stdout:Total 622 kB/s | 161 kB 00:00 2023-09-30T01:51:32.275 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-09-30T01:51:32.289 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-09-30T01:51:32.289 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-09-30T01:51:32.347 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-09-30T01:51:32.352 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-09-30T01:51:32.469 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-09-30T01:51:32.615 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T01:51:32.766 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:51:32.919 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:51:33.079 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T01:51:33.129 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:51:34.148 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T01:51:34.148 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:50 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:51:34.175 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:51:34.175 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:50 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:51:36.389 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-09-30T01:51:36.493 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:51:36.493 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-09-30T01:51:36.493 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:36.494 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:51:36.494 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:51:36.505 INFO:teuthology.orchestra.run.smithi007.stdout:Installed products updated. 2023-09-30T01:51:36.613 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr 2023-09-30T01:51:36.613 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:51:36.614 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-09-30T01:51:36.614 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:51:36.614 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:51:36.614 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:51:36.752 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-immutable-object-cache 2023-09-30T01:51:37.001 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:51:37.149 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:51:37.940 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:53 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:51:38.237 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:54 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:51:40.110 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:51:40.201 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:51:40.201 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:51:40.201 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-09-30T01:51:40.201 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:51:40.202 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-09-30T01:51:40.202 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-30T01:51:40.202 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:51:40.202 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-09-30T01:51:40.202 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:51:40.202 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-09-30T01:51:40.202 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:51:40.203 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 161 k 2023-09-30T01:51:40.203 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 490 k 2023-09-30T01:51:40.203 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-09-30T01:51:40.461 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 625 kB/s | 161 kB 00:00 2023-09-30T01:51:40.461 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:51:40.462 INFO:teuthology.orchestra.run.smithi007.stdout:Total 622 kB/s | 161 kB 00:00 2023-09-30T01:51:40.462 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-09-30T01:51:40.476 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-09-30T01:51:40.476 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-09-30T01:51:40.535 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-09-30T01:51:40.540 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-09-30T01:51:40.621 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:51:40.622 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-09-30T01:51:40.622 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:51:40.663 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-09-30T01:51:40.720 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-dashboard 2023-09-30T01:51:40.818 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T01:51:41.102 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:51:41.275 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T01:51:41.947 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:57 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:51:42.354 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T01:51:42.354 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:58 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:51:43.885 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:51:43.886 INFO:teuthology.orchestra.run.smithi050.stdout:=================================================================================================== 2023-09-30T01:51:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-09-30T01:51:43.887 INFO:teuthology.orchestra.run.smithi050.stdout:=================================================================================================== 2023-09-30T01:51:43.887 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-09-30T01:51:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-30T01:51:43.887 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-09-30T01:51:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-30T01:51:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-30T01:51:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-30T01:51:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-30T01:51:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-30T01:51:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:51:43.888 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-09-30T01:51:43.888 INFO:teuthology.orchestra.run.smithi050.stdout:=================================================================================================== 2023-09-30T01:51:43.888 INFO:teuthology.orchestra.run.smithi050.stdout:Install 6 Packages 2023-09-30T01:51:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:51:43.888 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 2.0 M 2023-09-30T01:51:43.888 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 8.0 M 2023-09-30T01:51:43.889 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-09-30T01:51:44.182 INFO:teuthology.orchestra.run.smithi050.stdout:(1/6): ceph-grafana-dashboards-17.2.6-1165.g93d 90 kB/s | 23 kB 00:00 2023-09-30T01:51:44.182 INFO:teuthology.orchestra.run.smithi050.stdout:(2/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 56 kB/s | 14 kB 00:00 2023-09-30T01:51:44.419 INFO:teuthology.orchestra.run.smithi050.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 142 kB/s | 34 kB 00:00 2023-09-30T01:51:44.497 INFO:teuthology.orchestra.run.smithi050.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 550 kB/s | 43 kB 00:00 2023-09-30T01:51:44.598 INFO:teuthology.orchestra.run.smithi050.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 473 kB/s | 196 kB 00:00 2023-09-30T01:51:45.005 INFO:teuthology.orchestra.run.smithi007.stdout:Installed products updated. 2023-09-30T01:51:45.110 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:51:45.110 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-09-30T01:51:45.110 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:45.110 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:51:45.111 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:51:45.199 INFO:teuthology.orchestra.run.smithi050.stdout:(6/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 1.3 MB/s | 1.7 MB 00:01 2023-09-30T01:51:45.201 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:51:45.202 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2023-09-30T01:51:45.204 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-09-30T01:51:45.214 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-09-30T01:51:45.215 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-09-30T01:51:45.265 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr 2023-09-30T01:51:45.306 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-09-30T01:51:45.310 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-09-30T01:51:45.610 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-09-30T01:51:45.653 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:51:45.819 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-09-30T01:51:46.056 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-09-30T01:51:46.265 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-09-30T01:51:46.411 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-30T01:51:46.824 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-30T01:51:47.093 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T01:51:47.288 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:03 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:51:47.370 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T01:51:47.370 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-30T01:51:47.370 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-30T01:51:47.371 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-30T01:51:47.371 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-09-30T01:51:47.371 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-09-30T01:51:48.410 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-09-30T01:51:48.410 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:04 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:51:48.972 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:51:49.513 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:51:49.515 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-09-30T01:51:49.515 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:51:49.878 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-dashboard 2023-09-30T01:51:50.261 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:51:52.006 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:08 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:51:52.364 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-09-30T01:51:52.472 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:51:52.472 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-09-30T01:51:52.472 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:51:52.472 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:51:52.472 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:51:52.472 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-30T01:51:52.472 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-30T01:51:52.472 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-30T01:51:52.473 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:51:52.473 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:51:52.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-30T01:51:52.993 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:51:53.985 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:51:53.986 INFO:teuthology.orchestra.run.smithi007.stdout:=================================================================================================== 2023-09-30T01:51:53.986 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-09-30T01:51:53.986 INFO:teuthology.orchestra.run.smithi007.stdout:=================================================================================================== 2023-09-30T01:51:53.986 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-09-30T01:51:53.986 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-30T01:51:53.986 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-09-30T01:51:53.986 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-30T01:51:53.986 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-30T01:51:53.987 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-30T01:51:53.987 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-30T01:51:53.987 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-30T01:51:53.987 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:51:53.987 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-09-30T01:51:53.987 INFO:teuthology.orchestra.run.smithi007.stdout:=================================================================================================== 2023-09-30T01:51:53.990 INFO:teuthology.orchestra.run.smithi007.stdout:Install 6 Packages 2023-09-30T01:51:53.990 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:51:53.990 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 2.0 M 2023-09-30T01:51:53.991 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 8.0 M 2023-09-30T01:51:53.991 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-09-30T01:51:54.069 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:10 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:51:54.249 INFO:teuthology.orchestra.run.smithi007.stdout:(1/6): ceph-grafana-dashboards-17.2.6-1165.g93d 124 kB/s | 23 kB 00:00 2023-09-30T01:51:54.252 INFO:teuthology.orchestra.run.smithi007.stdout:(2/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 76 kB/s | 14 kB 00:00 2023-09-30T01:51:54.309 INFO:teuthology.orchestra.run.smithi007.stdout:(3/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 6.9 MB/s | 1.7 MB 00:00 2023-09-30T01:51:54.355 INFO:teuthology.orchestra.run.smithi007.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 941 kB/s | 43 kB 00:00 2023-09-30T01:51:54.447 INFO:teuthology.orchestra.run.smithi007.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 170 kB/s | 34 kB 00:00 2023-09-30T01:51:54.553 INFO:teuthology.orchestra.run.smithi007.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 663 kB/s | 196 kB 00:00 2023-09-30T01:51:54.553 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:51:54.553 INFO:teuthology.orchestra.run.smithi007.stdout:Total 3.5 MB/s | 2.0 MB 00:00 2023-09-30T01:51:54.556 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-09-30T01:51:54.564 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-09-30T01:51:54.565 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-09-30T01:51:54.651 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-09-30T01:51:54.656 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-09-30T01:51:54.967 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-09-30T01:51:55.201 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-09-30T01:51:55.443 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-09-30T01:51:55.610 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-09-30T01:51:55.773 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-30T01:51:56.135 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-30T01:51:56.398 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:51:56.400 INFO:teuthology.orchestra.run.smithi050.stdout:====================================================================================================================== 2023-09-30T01:51:56.400 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-09-30T01:51:56.400 INFO:teuthology.orchestra.run.smithi050.stdout:====================================================================================================================== 2023-09-30T01:51:56.400 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-09-30T01:51:56.400 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-30T01:51:56.400 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-09-30T01:51:56.400 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-09-30T01:51:56.401 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-09-30T01:51:56.401 INFO:teuthology.orchestra.run.smithi050.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-09-30T01:51:56.401 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-09-30T01:51:56.401 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-09-30T01:51:56.401 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-09-30T01:51:56.401 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-09-30T01:51:56.401 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:51:56.401 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-09-30T01:51:56.401 INFO:teuthology.orchestra.run.smithi050.stdout:====================================================================================================================== 2023-09-30T01:51:56.401 INFO:teuthology.orchestra.run.smithi050.stdout:Install 8 Packages 2023-09-30T01:51:56.401 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:51:56.402 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T01:51:56.402 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 36 M 2023-09-30T01:51:56.402 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 199 M 2023-09-30T01:51:56.403 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-09-30T01:51:56.451 INFO:teuthology.orchestra.run.smithi050.stdout:(1/8): libquadmath-8.4.1-1.el8.x86_64.rpm 3.4 MB/s | 169 kB 00:00 2023-09-30T01:51:56.478 INFO:teuthology.orchestra.run.smithi050.stdout:(2/8): libgfortran-8.4.1-1.el8.x86_64.rpm 8.3 MB/s | 639 kB 00:00 2023-09-30T01:51:56.528 INFO:teuthology.orchestra.run.smithi050.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 4.4 MB/s | 225 kB 00:00 2023-09-30T01:51:56.621 INFO:teuthology.orchestra.run.smithi050.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 41 MB/s | 3.7 MB 00:00 2023-09-30T01:51:56.677 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T01:51:56.678 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-30T01:51:56.678 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-30T01:51:56.678 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-30T01:51:56.678 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-09-30T01:51:56.678 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-09-30T01:51:56.765 INFO:teuthology.orchestra.run.smithi050.stdout:(5/8): python3-scipy-1.0.0-20.module+el8.1.0+33 46 MB/s | 15 MB 00:00 2023-09-30T01:51:56.832 INFO:teuthology.orchestra.run.smithi050.stdout:(6/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2023-09-30T01:51:56.863 INFO:teuthology.orchestra.run.smithi050.stdout:(7/8): openblas-0.3.12-1.el8.x86_64.rpm 47 MB/s | 4.6 MB 00:00 2023-09-30T01:51:56.925 INFO:teuthology.orchestra.run.smithi050.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.6-116 14 MB/s | 7.4 MB 00:00 2023-09-30T01:51:56.926 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:51:56.927 INFO:teuthology.orchestra.run.smithi050.stdout:Total 68 MB/s | 36 MB 00:00 2023-09-30T01:51:56.937 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-09-30T01:51:56.958 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-09-30T01:51:56.958 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-09-30T01:51:57.381 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-09-30T01:51:57.386 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-09-30T01:51:57.720 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-09-30T01:51:57.720 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:13 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:51:58.035 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-09-30T01:51:58.244 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2023-09-30T01:51:58.389 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2023-09-30T01:51:58.544 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2023-09-30T01:51:59.123 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2023-09-30T01:51:59.327 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-09-30T01:51:59.888 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-09-30T01:51:59.974 INFO:teuthology.orchestra.run.smithi007.stdout:Installed products updated. 2023-09-30T01:52:00.083 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:00.083 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-09-30T01:52:00.083 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:52:00.083 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:52:00.083 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:52:00.083 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-30T01:52:00.083 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-30T01:52:00.083 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-30T01:52:00.084 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:00.084 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:52:00.119 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2023-09-30T01:52:00.392 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-30T01:52:00.778 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:52:00.900 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2023-09-30T01:52:01.434 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2023-09-30T01:52:01.668 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-09-30T01:52:02.641 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:18 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:03.835 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-09-30T01:52:04.631 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:52:04.632 INFO:teuthology.orchestra.run.smithi007.stdout:====================================================================================================================== 2023-09-30T01:52:04.632 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-09-30T01:52:04.632 INFO:teuthology.orchestra.run.smithi007.stdout:====================================================================================================================== 2023-09-30T01:52:04.632 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-09-30T01:52:04.632 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-30T01:52:04.633 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-09-30T01:52:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-09-30T01:52:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-09-30T01:52:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-09-30T01:52:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-09-30T01:52:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-09-30T01:52:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-09-30T01:52:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-09-30T01:52:04.633 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:04.633 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-09-30T01:52:04.633 INFO:teuthology.orchestra.run.smithi007.stdout:====================================================================================================================== 2023-09-30T01:52:04.633 INFO:teuthology.orchestra.run.smithi007.stdout:Install 8 Packages 2023-09-30T01:52:04.634 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:04.634 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 36 M 2023-09-30T01:52:04.634 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 199 M 2023-09-30T01:52:04.634 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-09-30T01:52:04.786 INFO:teuthology.orchestra.run.smithi007.stdout:(1/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 1.5 MB/s | 225 kB 00:00 2023-09-30T01:52:04.962 INFO:teuthology.orchestra.run.smithi007.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 21 MB/s | 3.7 MB 00:00 2023-09-30T01:52:05.141 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2023-09-30T01:52:05.152 INFO:teuthology.orchestra.run.smithi007.stdout:(3/8): python3-scipy-1.0.0-20.module+el8.1.0+33 28 MB/s | 15 MB 00:00 2023-09-30T01:52:05.294 INFO:teuthology.orchestra.run.smithi007.stdout:(4/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2023-09-30T01:52:05.368 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T01:52:05.378 INFO:teuthology.orchestra.run.smithi007.stdout:(5/8): openblas-0.3.12-1.el8.x86_64.rpm 20 MB/s | 4.6 MB 00:00 2023-09-30T01:52:05.416 INFO:teuthology.orchestra.run.smithi007.stdout:(6/8): libquadmath-8.4.1-1.el8.x86_64.rpm 1.4 MB/s | 169 kB 00:00 2023-09-30T01:52:05.441 INFO:teuthology.orchestra.run.smithi007.stdout:(7/8): ceph-mgr-diskprediction-local-17.2.6-116 9.2 MB/s | 7.4 MB 00:00 2023-09-30T01:52:05.530 INFO:teuthology.orchestra.run.smithi007.stdout:(8/8): libgfortran-8.4.1-1.el8.x86_64.rpm 4.1 MB/s | 639 kB 00:00 2023-09-30T01:52:05.531 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:05.532 INFO:teuthology.orchestra.run.smithi007.stdout:Total 40 MB/s | 36 MB 00:00 2023-09-30T01:52:05.542 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-09-30T01:52:05.563 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-09-30T01:52:05.563 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-09-30T01:52:05.684 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T01:52:05.684 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-30T01:52:05.684 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 2/8 2023-09-30T01:52:05.684 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 3/8 2023-09-30T01:52:05.684 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 4/8 2023-09-30T01:52:05.685 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2023-09-30T01:52:05.685 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 6/8 2023-09-30T01:52:05.685 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 7/8 2023-09-30T01:52:06.018 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-09-30T01:52:06.022 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-09-30T01:52:06.678 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-09-30T01:52:06.758 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 8/8 2023-09-30T01:52:06.758 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:22 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:06.863 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2023-09-30T01:52:07.003 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2023-09-30T01:52:07.163 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2023-09-30T01:52:07.743 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2023-09-30T01:52:07.955 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-09-30T01:52:08.516 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2023-09-30T01:52:08.738 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2023-09-30T01:52:09.002 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-09-30T01:52:09.111 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:09.111 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-09-30T01:52:09.111 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:52:09.112 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-09-30T01:52:09.112 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-09-30T01:52:09.112 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-0.3.12-1.el8.x86_64 2023-09-30T01:52:09.112 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-09-30T01:52:09.112 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-09-30T01:52:09.112 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-09-30T01:52:09.112 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-09-30T01:52:09.112 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:09.112 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:52:09.286 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-rook 2023-09-30T01:52:09.525 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2023-09-30T01:52:09.674 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:52:09.861 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2023-09-30T01:52:10.071 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-09-30T01:52:10.488 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:26 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:12.265 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2023-09-30T01:52:12.455 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:52:12.457 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================================== 2023-09-30T01:52:12.457 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-09-30T01:52:12.457 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================================== 2023-09-30T01:52:12.457 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-09-30T01:52:12.458 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-30T01:52:12.458 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-09-30T01:52:12.458 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-30T01:52:12.458 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-30T01:52:12.458 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-30T01:52:12.458 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-09-30T01:52:12.458 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-09-30T01:52:12.458 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-30T01:52:12.458 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-09-30T01:52:12.458 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-09-30T01:52:12.458 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-09-30T01:52:12.459 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-30T01:52:12.459 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-30T01:52:12.459 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-30T01:52:12.459 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:12.459 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-09-30T01:52:12.459 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================================== 2023-09-30T01:52:12.459 INFO:teuthology.orchestra.run.smithi050.stdout:Install 13 Packages 2023-09-30T01:52:12.459 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:12.460 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 1.9 M 2023-09-30T01:52:12.460 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 25 M 2023-09-30T01:52:12.460 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-09-30T01:52:12.986 INFO:teuthology.orchestra.run.smithi050.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 76 kB/s | 16 kB 00:00 2023-09-30T01:52:12.995 INFO:teuthology.orchestra.run.smithi050.stdout:(2/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 212 kB/s | 47 kB 00:00 2023-09-30T01:52:13.000 INFO:teuthology.orchestra.run.smithi050.stdout:(3/13): python3-cachetools-3.1.1-4.el8.noarch.r 144 kB/s | 33 kB 00:00 2023-09-30T01:52:13.109 INFO:teuthology.orchestra.run.smithi050.stdout:(4/13): python3-rsa-4.9-2.el8.noarch.rpm 588 kB/s | 64 kB 00:00 2023-09-30T01:52:13.122 INFO:teuthology.orchestra.run.smithi050.stdout:(5/13): python3-google-auth-1.1.1-10.el8.noarch 598 kB/s | 81 kB 00:00 2023-09-30T01:52:13.170 INFO:teuthology.orchestra.run.smithi050.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.2 MB/s | 155 kB 00:00 2023-09-30T01:52:13.178 INFO:teuthology.orchestra.run.smithi050.stdout:(7/13): python3-websocket-client-0.56.0-5.el8.n 893 kB/s | 61 kB 00:00 2023-09-30T01:52:13.205 INFO:teuthology.orchestra.run.smithi050.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.2 MB/s | 43 kB 00:00 2023-09-30T01:52:13.224 INFO:teuthology.orchestra.run.smithi050.stdout:(9/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 577 kB/s | 27 kB 00:00 2023-09-30T01:52:13.233 INFO:teuthology.orchestra.run.smithi050.stdout:(10/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 4.5 MB/s | 126 kB 00:00 2023-09-30T01:52:13.250 INFO:teuthology.orchestra.run.smithi050.stdout:(11/13): python3-pyasn1-modules-0.3.7-6.el8.noa 4.2 MB/s | 110 kB 00:00 2023-09-30T01:52:13.259 INFO:teuthology.orchestra.run.smithi050.stdout:(12/13): python3-jsonpointer-1.10-11.el8.noarch 773 kB/s | 20 kB 00:00 2023-09-30T01:52:13.461 INFO:teuthology.orchestra.run.smithi050.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.5 MB/s | 1.2 MB 00:00 2023-09-30T01:52:13.463 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:13.464 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.9 MB/s | 1.9 MB 00:01 2023-09-30T01:52:13.471 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-09-30T01:52:13.488 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-09-30T01:52:13.488 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-09-30T01:52:13.582 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-09-30T01:52:13.587 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-09-30T01:52:13.623 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2023-09-30T01:52:13.870 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T01:52:14.146 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-09-30T01:52:14.206 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T01:52:14.207 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-30T01:52:14.207 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 2/8 2023-09-30T01:52:14.207 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 3/8 2023-09-30T01:52:14.207 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2023-09-30T01:52:14.207 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 5/8 2023-09-30T01:52:14.207 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 6/8 2023-09-30T01:52:14.207 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 7/8 2023-09-30T01:52:14.458 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-30T01:52:14.720 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-09-30T01:52:14.925 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-09-30T01:52:15.177 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2023-09-30T01:52:15.396 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 8/8 2023-09-30T01:52:15.397 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:31 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:15.423 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2023-09-30T01:52:15.738 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-30T01:52:16.004 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-30T01:52:16.278 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-09-30T01:52:16.489 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-30T01:52:16.762 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-09-30T01:52:17.738 INFO:teuthology.orchestra.run.smithi007.stdout:Installed products updated. 2023-09-30T01:52:17.848 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:17.848 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-09-30T01:52:17.848 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:52:17.848 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-09-30T01:52:17.848 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-09-30T01:52:17.848 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-0.3.12-1.el8.x86_64 2023-09-30T01:52:17.848 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-09-30T01:52:17.848 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-09-30T01:52:17.849 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-09-30T01:52:17.849 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-09-30T01:52:17.849 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:17.849 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:52:17.994 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-rook 2023-09-30T01:52:18.146 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-09-30T01:52:18.382 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:52:18.479 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-09-30T01:52:18.625 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T01:52:18.892 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T01:52:18.892 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-30T01:52:18.892 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-09-30T01:52:18.892 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-09-30T01:52:18.892 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-09-30T01:52:18.892 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-09-30T01:52:18.892 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-09-30T01:52:18.892 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-09-30T01:52:18.893 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-30T01:52:18.893 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-30T01:52:18.893 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 10/13 2023-09-30T01:52:18.893 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 11/13 2023-09-30T01:52:18.893 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 12/13 2023-09-30T01:52:19.331 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:35 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:19.939 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 13/13 2023-09-30T01:52:19.940 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:35 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:21.219 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:52:21.221 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================================== 2023-09-30T01:52:21.221 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-09-30T01:52:21.221 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================================== 2023-09-30T01:52:21.221 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-09-30T01:52:21.221 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-30T01:52:21.221 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-09-30T01:52:21.221 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-30T01:52:21.221 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-30T01:52:21.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-30T01:52:21.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-09-30T01:52:21.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-09-30T01:52:21.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-30T01:52:21.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-09-30T01:52:21.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-09-30T01:52:21.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-09-30T01:52:21.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-09-30T01:52:21.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-30T01:52:21.222 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-30T01:52:21.222 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:21.223 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-09-30T01:52:21.223 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================================== 2023-09-30T01:52:21.223 INFO:teuthology.orchestra.run.smithi007.stdout:Install 13 Packages 2023-09-30T01:52:21.223 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:21.223 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 1.9 M 2023-09-30T01:52:21.223 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 25 M 2023-09-30T01:52:21.224 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-09-30T01:52:21.430 INFO:teuthology.orchestra.run.smithi007.stdout:(1/13): python3-cachetools-3.1.1-4.el8.noarch.r 244 kB/s | 33 kB 00:00 2023-09-30T01:52:21.437 INFO:teuthology.orchestra.run.smithi007.stdout:(2/13): python3-certifi-2018.10.15-7.el8.noarch 114 kB/s | 16 kB 00:00 2023-09-30T01:52:21.494 INFO:teuthology.orchestra.run.smithi007.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 1.2 MB/s | 81 kB 00:00 2023-09-30T01:52:21.522 INFO:teuthology.orchestra.run.smithi007.stdout:(4/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 208 kB/s | 47 kB 00:00 2023-09-30T01:52:21.528 INFO:teuthology.orchestra.run.smithi007.stdout:(5/13): python3-rsa-4.9-2.el8.noarch.rpm 1.8 MB/s | 64 kB 00:00 2023-09-30T01:52:21.632 INFO:teuthology.orchestra.run.smithi007.stdout:(6/13): python3-kubernetes-11.0.0-6.el8.noarch. 6.0 MB/s | 1.2 MB 00:00 2023-09-30T01:52:21.664 INFO:teuthology.orchestra.run.smithi007.stdout:(7/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 195 kB/s | 27 kB 00:00 2023-09-30T01:52:21.692 INFO:teuthology.orchestra.run.smithi007.stdout:(8/13): python3-websocket-client-0.56.0-5.el8.n 361 kB/s | 61 kB 00:00 2023-09-30T01:52:21.782 INFO:teuthology.orchestra.run.smithi007.stdout:(9/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 843 kB/s | 126 kB 00:00 2023-09-30T01:52:21.805 INFO:teuthology.orchestra.run.smithi007.stdout:(10/13): python3-pyasn1-modules-0.3.7-6.el8.noa 783 kB/s | 110 kB 00:00 2023-09-30T01:52:21.817 INFO:teuthology.orchestra.run.smithi007.stdout:(11/13): python3-jsonpointer-1.10-11.el8.noarch 160 kB/s | 20 kB 00:00 2023-09-30T01:52:21.888 INFO:teuthology.orchestra.run.smithi007.stdout:(12/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 1.4 MB/s | 155 kB 00:00 2023-09-30T01:52:21.907 INFO:teuthology.orchestra.run.smithi007.stdout:(13/13): python3-requests-oauthlib-1.0.0-1.el8. 419 kB/s | 43 kB 00:00 2023-09-30T01:52:21.910 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:21.911 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.8 MB/s | 1.9 MB 00:00 2023-09-30T01:52:21.917 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-09-30T01:52:21.933 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-09-30T01:52:21.933 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-09-30T01:52:22.033 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-09-30T01:52:22.038 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-09-30T01:52:22.558 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-09-30T01:52:22.828 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-30T01:52:23.072 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-09-30T01:52:23.342 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-09-30T01:52:23.602 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2023-09-30T01:52:23.685 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-09-30T01:52:23.794 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:23.794 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-09-30T01:52:23.794 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:52:23.794 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-30T01:52:23.794 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-30T01:52:23.794 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-30T01:52:23.794 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-30T01:52:23.794 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-30T01:52:23.794 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-30T01:52:23.795 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-30T01:52:23.795 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-30T01:52:23.795 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-30T01:52:23.795 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-30T01:52:23.795 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-30T01:52:23.795 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-30T01:52:23.795 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:23.795 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:52:23.877 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2023-09-30T01:52:23.971 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-cephadm 2023-09-30T01:52:24.112 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2023-09-30T01:52:24.368 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:52:24.473 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2023-09-30T01:52:24.738 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-09-30T01:52:24.982 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-30T01:52:25.252 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:41 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:25.255 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-09-30T01:52:26.528 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-09-30T01:52:26.881 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-09-30T01:52:27.010 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T01:52:27.258 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:52:27.259 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================================== 2023-09-30T01:52:27.259 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-09-30T01:52:27.259 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================================== 2023-09-30T01:52:27.259 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-09-30T01:52:27.259 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-30T01:52:27.259 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-09-30T01:52:27.259 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-30T01:52:27.260 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-09-30T01:52:27.260 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-09-30T01:52:27.260 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-30T01:52:27.260 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:27.260 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-09-30T01:52:27.260 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================================== 2023-09-30T01:52:27.260 INFO:teuthology.orchestra.run.smithi050.stdout:Install 5 Packages 2023-09-30T01:52:27.260 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:27.261 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 6.0 M 2023-09-30T01:52:27.261 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 26 M 2023-09-30T01:52:27.261 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-09-30T01:52:27.286 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T01:52:27.287 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-30T01:52:27.287 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-09-30T01:52:27.287 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-09-30T01:52:27.287 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-09-30T01:52:27.287 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-09-30T01:52:27.287 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-09-30T01:52:27.287 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-09-30T01:52:27.287 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 8/13 2023-09-30T01:52:27.287 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 9/13 2023-09-30T01:52:27.288 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 10/13 2023-09-30T01:52:27.288 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-30T01:52:27.288 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 12/13 2023-09-30T01:52:27.461 INFO:teuthology.orchestra.run.smithi050.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 427 kB/s | 60 kB 00:00 2023-09-30T01:52:27.550 INFO:teuthology.orchestra.run.smithi050.stdout:(2/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.0 MB/s | 476 kB 00:00 2023-09-30T01:52:27.567 INFO:teuthology.orchestra.run.smithi050.stdout:(3/5): python3-babel-2.5.1-5.el8.noarch.rpm 45 MB/s | 4.8 MB 00:00 2023-09-30T01:52:27.577 INFO:teuthology.orchestra.run.smithi050.stdout:(4/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 471 kB/s | 121 kB 00:00 2023-09-30T01:52:27.598 INFO:teuthology.orchestra.run.smithi050.stdout:(5/5): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 11 MB/s | 538 kB 00:00 2023-09-30T01:52:27.599 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:27.599 INFO:teuthology.orchestra.run.smithi050.stdout:Total 18 MB/s | 6.0 MB 00:00 2023-09-30T01:52:27.602 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-09-30T01:52:27.613 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-09-30T01:52:27.614 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-09-30T01:52:27.711 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-09-30T01:52:27.716 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-09-30T01:52:28.476 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 13/13 2023-09-30T01:52:28.477 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:44 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:28.805 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-09-30T01:52:29.197 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/5 2023-09-30T01:52:29.454 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/5 2023-09-30T01:52:29.822 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-30T01:52:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-30T01:52:30.264 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T01:52:30.466 INFO:teuthology.orchestra.run.smithi007.stdout:Installed products updated. 2023-09-30T01:52:30.550 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T01:52:30.550 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-30T01:52:30.550 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-30T01:52:30.550 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-30T01:52:30.550 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 4/5 2023-09-30T01:52:30.575 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:30.575 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-09-30T01:52:30.575 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:52:30.575 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-30T01:52:30.575 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-30T01:52:30.575 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-30T01:52:30.575 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-30T01:52:30.575 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-30T01:52:30.575 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-30T01:52:30.576 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-30T01:52:30.576 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-30T01:52:30.576 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-30T01:52:30.576 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-30T01:52:30.576 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-30T01:52:30.576 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-30T01:52:30.576 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:30.576 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:52:30.715 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-cephadm 2023-09-30T01:52:31.113 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:52:31.570 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 5/5 2023-09-30T01:52:31.570 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:47 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:32.014 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:48 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:33.582 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-09-30T01:52:33.690 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:33.690 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-09-30T01:52:33.691 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:52:33.691 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-30T01:52:33.691 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel-2.5.1-5.el8.noarch 2023-09-30T01:52:33.691 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-09-30T01:52:33.691 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-30T01:52:33.691 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:33.691 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:52:33.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-fuse 2023-09-30T01:52:33.963 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:52:33.964 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================================== 2023-09-30T01:52:33.964 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-09-30T01:52:33.964 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================================== 2023-09-30T01:52:33.964 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-09-30T01:52:33.964 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-30T01:52:33.965 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-09-30T01:52:33.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-30T01:52:33.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-09-30T01:52:33.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-09-30T01:52:33.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-30T01:52:33.965 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:33.965 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-09-30T01:52:33.965 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================================== 2023-09-30T01:52:33.965 INFO:teuthology.orchestra.run.smithi007.stdout:Install 5 Packages 2023-09-30T01:52:33.965 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:33.966 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 6.0 M 2023-09-30T01:52:33.966 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 26 M 2023-09-30T01:52:33.966 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-09-30T01:52:34.210 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:52:34.485 INFO:teuthology.orchestra.run.smithi007.stdout:(1/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 788 kB/s | 121 kB 00:00 2023-09-30T01:52:34.589 INFO:teuthology.orchestra.run.smithi007.stdout:(2/5): python3-babel-2.5.1-5.el8.noarch.rpm 46 MB/s | 4.8 MB 00:00 2023-09-30T01:52:34.611 INFO:teuthology.orchestra.run.smithi007.stdout:(3/5): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 25 MB/s | 538 kB 00:00 2023-09-30T01:52:34.816 INFO:teuthology.orchestra.run.smithi007.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 124 kB/s | 60 kB 00:00 2023-09-30T01:52:35.056 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:51 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:35.105 INFO:teuthology.orchestra.run.smithi007.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 615 kB/s | 476 kB 00:00 2023-09-30T01:52:35.106 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:35.107 INFO:teuthology.orchestra.run.smithi007.stdout:Total 5.2 MB/s | 6.0 MB 00:01 2023-09-30T01:52:35.110 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-09-30T01:52:35.122 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-09-30T01:52:35.122 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-09-30T01:52:35.224 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-09-30T01:52:35.229 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-09-30T01:52:36.321 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-09-30T01:52:36.980 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:52:36.981 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:52:36.981 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-09-30T01:52:36.981 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:52:36.981 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-09-30T01:52:36.981 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse 2023-09-30T01:52:36.981 INFO:teuthology.orchestra.run.smithi050.stdout: x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-30T01:52:36.982 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2023-09-30T01:52:36.982 INFO:teuthology.orchestra.run.smithi050.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-09-30T01:52:36.982 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:36.982 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-09-30T01:52:36.982 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:52:36.982 INFO:teuthology.orchestra.run.smithi050.stdout:Install 2 Packages 2023-09-30T01:52:36.982 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:36.983 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 957 k 2023-09-30T01:52:36.983 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 3.0 M 2023-09-30T01:52:36.983 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-09-30T01:52:37.016 INFO:teuthology.orchestra.run.smithi050.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.4 MB/s | 83 kB 00:00 2023-09-30T01:52:37.017 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/5 2023-09-30T01:52:37.299 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/5 2023-09-30T01:52:37.350 INFO:teuthology.orchestra.run.smithi050.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 2.3 MB/s | 874 kB 00:00 2023-09-30T01:52:37.351 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:37.351 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.5 MB/s | 957 kB 00:00 2023-09-30T01:52:37.353 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-09-30T01:52:37.376 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-09-30T01:52:37.376 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-09-30T01:52:37.465 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-09-30T01:52:37.470 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-09-30T01:52:37.639 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-09-30T01:52:37.680 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-30T01:52:37.941 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2023-09-30T01:52:37.974 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-30T01:52:38.106 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T01:52:38.133 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T01:52:38.440 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T01:52:38.440 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-30T01:52:38.440 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-30T01:52:38.440 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-09-30T01:52:38.440 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 4/5 2023-09-30T01:52:38.556 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T01:52:38.556 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T01:52:39.565 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 5/5 2023-09-30T01:52:39.565 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:55 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:40.171 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2023-09-30T01:52:40.171 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:56 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:41.885 INFO:teuthology.orchestra.run.smithi007.stdout:Installed products updated. 2023-09-30T01:52:41.993 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:41.993 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-09-30T01:52:41.993 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:52:41.993 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-30T01:52:41.993 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel-2.5.1-5.el8.noarch 2023-09-30T01:52:41.993 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-09-30T01:52:41.994 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-30T01:52:41.994 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:41.994 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:52:42.209 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-09-30T01:52:42.315 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:42.315 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-09-30T01:52:42.315 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2023-09-30T01:52:42.316 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:42.316 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:52:42.442 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-volume 2023-09-30T01:52:42.838 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:52:43.309 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-fuse 2023-09-30T01:52:43.705 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:52:43.715 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:02:59 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:44.673 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:00 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:45.722 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:52:45.722 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:52:45.722 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2023-09-30T01:52:45.723 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:52:45.723 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-09-30T01:52:45.723 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-30T01:52:45.723 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:45.723 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-09-30T01:52:45.723 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:52:45.723 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-09-30T01:52:45.723 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:45.723 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 267 k 2023-09-30T01:52:45.723 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 1.3 M 2023-09-30T01:52:45.723 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-09-30T01:52:46.029 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 874 kB/s | 267 kB 00:00 2023-09-30T01:52:46.030 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:46.030 INFO:teuthology.orchestra.run.smithi050.stdout:Total 869 kB/s | 267 kB 00:00 2023-09-30T01:52:46.030 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-09-30T01:52:46.042 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-09-30T01:52:46.042 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-09-30T01:52:46.070 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-09-30T01:52:46.075 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-09-30T01:52:46.266 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-09-30T01:52:46.546 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:52:46.569 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:52:46.571 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:52:46.571 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-09-30T01:52:46.571 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:52:46.571 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-09-30T01:52:46.571 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse 2023-09-30T01:52:46.571 INFO:teuthology.orchestra.run.smithi007.stdout: x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-30T01:52:46.571 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-09-30T01:52:46.571 INFO:teuthology.orchestra.run.smithi007.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-09-30T01:52:46.571 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:46.571 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-09-30T01:52:46.572 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:52:46.572 INFO:teuthology.orchestra.run.smithi007.stdout:Install 2 Packages 2023-09-30T01:52:46.572 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:46.572 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 957 k 2023-09-30T01:52:46.572 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 3.0 M 2023-09-30T01:52:46.572 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-09-30T01:52:46.614 INFO:teuthology.orchestra.run.smithi007.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 1.9 MB/s | 83 kB 00:00 2023-09-30T01:52:46.939 INFO:teuthology.orchestra.run.smithi007.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 2.3 MB/s | 874 kB 00:00 2023-09-30T01:52:46.939 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:46.940 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.5 MB/s | 957 kB 00:00 2023-09-30T01:52:46.941 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-09-30T01:52:46.964 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-09-30T01:52:46.964 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-09-30T01:52:46.984 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:52:47.053 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-09-30T01:52:47.057 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-09-30T01:52:47.239 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-09-30T01:52:47.557 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2023-09-30T01:52:47.715 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T01:52:48.022 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:52:48.022 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:04 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:48.132 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T01:52:48.133 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T01:52:49.206 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2023-09-30T01:52:49.206 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:05 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:50.285 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-09-30T01:52:50.392 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:50.393 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-09-30T01:52:50.393 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:52:50.393 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:50.393 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:52:50.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados-devel 2023-09-30T01:52:50.920 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:52:51.417 INFO:teuthology.orchestra.run.smithi007.stdout:Installed products updated. 2023-09-30T01:52:51.524 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:51.524 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-09-30T01:52:51.524 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2023-09-30T01:52:51.524 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:51.524 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:52:51.659 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-volume 2023-09-30T01:52:51.774 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:07 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:52.057 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:52:53.027 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:09 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:53.793 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:52:53.793 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:52:53.793 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2023-09-30T01:52:53.793 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:52:53.793 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-09-30T01:52:53.794 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-30T01:52:53.794 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:53.794 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-09-30T01:52:53.794 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:52:53.794 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-09-30T01:52:53.794 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:53.795 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 130 k 2023-09-30T01:52:53.795 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 464 k 2023-09-30T01:52:53.795 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-09-30T01:52:54.090 INFO:teuthology.orchestra.run.smithi050.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 440 kB/s | 130 kB 00:00 2023-09-30T01:52:54.091 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:54.091 INFO:teuthology.orchestra.run.smithi050.stdout:Total 437 kB/s | 130 kB 00:00 2023-09-30T01:52:54.091 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-09-30T01:52:54.099 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-09-30T01:52:54.099 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-09-30T01:52:54.160 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-09-30T01:52:54.164 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-09-30T01:52:54.290 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-09-30T01:52:54.511 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:54.792 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:55.109 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:52:55.109 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:52:55.109 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-09-30T01:52:55.109 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:52:55.110 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-09-30T01:52:55.110 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-30T01:52:55.110 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:55.110 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-09-30T01:52:55.110 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:52:55.110 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-09-30T01:52:55.110 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:55.111 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 267 k 2023-09-30T01:52:55.111 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.3 M 2023-09-30T01:52:55.111 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-09-30T01:52:55.285 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 1.5 MB/s | 267 kB 00:00 2023-09-30T01:52:55.285 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:55.286 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.5 MB/s | 267 kB 00:00 2023-09-30T01:52:55.286 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-09-30T01:52:55.297 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-09-30T01:52:55.297 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-09-30T01:52:55.320 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-09-30T01:52:55.325 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-09-30T01:52:55.515 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-09-30T01:52:55.774 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:55.774 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:11 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:55.797 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:52:56.226 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:52:57.313 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:52:57.313 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:13 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:57.965 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-09-30T01:52:58.071 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:58.072 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-09-30T01:52:58.072 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:52:58.072 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:52:58.072 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:52:58.198 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs2 2023-09-30T01:52:58.594 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:52:59.717 INFO:teuthology.orchestra.run.smithi007.stdout:Installed products updated. 2023-09-30T01:52:59.727 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:15 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:52:59.823 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:59.823 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-09-30T01:52:59.823 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:52:59.823 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:52:59.823 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:52:59.954 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados-devel 2023-09-30T01:53:00.355 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:53:01.229 INFO:teuthology.orchestra.run.smithi050.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:53:01.405 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:17 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:01.729 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:53:01.730 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-09-30T01:53:01.730 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:53:01.843 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs-devel 2023-09-30T01:53:02.237 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:53:03.149 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:19 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:03.395 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:53:03.396 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:53:03.396 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-09-30T01:53:03.396 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:53:03.396 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-09-30T01:53:03.396 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-30T01:53:03.396 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:53:03.396 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-09-30T01:53:03.396 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:53:03.396 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-09-30T01:53:03.396 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:53:03.397 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 130 k 2023-09-30T01:53:03.397 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 464 k 2023-09-30T01:53:03.397 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-09-30T01:53:03.786 INFO:teuthology.orchestra.run.smithi007.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 333 kB/s | 130 kB 00:00 2023-09-30T01:53:03.787 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:53:03.787 INFO:teuthology.orchestra.run.smithi007.stdout:Total 332 kB/s | 130 kB 00:00 2023-09-30T01:53:03.788 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-09-30T01:53:03.795 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-09-30T01:53:03.796 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-09-30T01:53:03.857 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-09-30T01:53:03.862 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-09-30T01:53:03.994 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-09-30T01:53:04.178 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:04.485 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:05.137 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:53:05.138 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:53:05.138 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2023-09-30T01:53:05.138 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:53:05.139 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-09-30T01:53:05.139 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-30T01:53:05.139 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:53:05.139 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-09-30T01:53:05.139 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:53:05.139 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-09-30T01:53:05.139 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:53:05.139 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 25 k 2023-09-30T01:53:05.139 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 109 k 2023-09-30T01:53:05.139 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-09-30T01:53:05.354 INFO:teuthology.orchestra.run.smithi050.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 116 kB/s | 25 kB 00:00 2023-09-30T01:53:05.354 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:53:05.355 INFO:teuthology.orchestra.run.smithi050.stdout:Total 115 kB/s | 25 kB 00:00 2023-09-30T01:53:05.355 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-09-30T01:53:05.359 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-09-30T01:53:05.359 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-09-30T01:53:05.361 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-09-30T01:53:05.366 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-09-30T01:53:05.428 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-09-30T01:53:05.576 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:05.592 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:05.592 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:21 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:05.863 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:06.824 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:06.824 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:22 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:07.722 INFO:teuthology.orchestra.run.smithi007.stdout:Installed products updated. 2023-09-30T01:53:07.827 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:53:07.827 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-09-30T01:53:07.827 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:53:07.827 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:53:07.827 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:53:07.991 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs2 2023-09-30T01:53:08.390 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:53:08.980 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-09-30T01:53:09.086 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:53:09.086 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-09-30T01:53:09.086 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:53:09.086 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:53:09.087 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:53:09.203 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados2 2023-09-30T01:53:09.356 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:25 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:09.598 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:53:10.460 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:26 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:11.092 INFO:teuthology.orchestra.run.smithi007.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:53:11.595 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:53:11.597 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-09-30T01:53:11.597 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:53:11.698 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs-devel 2023-09-30T01:53:11.903 INFO:teuthology.orchestra.run.smithi050.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:53:12.093 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:53:12.396 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:53:12.397 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-09-30T01:53:12.397 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:53:12.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librbd1 2023-09-30T01:53:12.895 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:53:13.806 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:29 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:14.605 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:30 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:15.707 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:53:15.708 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:53:15.708 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-09-30T01:53:15.708 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:53:15.708 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-09-30T01:53:15.708 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-30T01:53:15.708 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:53:15.709 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-09-30T01:53:15.709 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:53:15.709 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-09-30T01:53:15.709 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:53:15.709 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 25 k 2023-09-30T01:53:15.709 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 109 k 2023-09-30T01:53:15.709 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-09-30T01:53:16.033 INFO:teuthology.orchestra.run.smithi050.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:53:16.113 INFO:teuthology.orchestra.run.smithi007.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 61 kB/s | 25 kB 00:00 2023-09-30T01:53:16.113 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:53:16.114 INFO:teuthology.orchestra.run.smithi007.stdout:Total 61 kB/s | 25 kB 00:00 2023-09-30T01:53:16.114 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-09-30T01:53:16.118 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-09-30T01:53:16.118 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-09-30T01:53:16.120 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-09-30T01:53:16.125 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-09-30T01:53:16.175 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-09-30T01:53:16.360 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:16.526 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:53:16.527 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-09-30T01:53:16.527 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:53:16.618 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:16.636 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rados 2023-09-30T01:53:17.029 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:53:17.688 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:17.688 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:33 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:18.266 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:34 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:19.802 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:53:19.896 INFO:teuthology.orchestra.run.smithi007.stdout:Installed products updated. 2023-09-30T01:53:20.004 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:53:20.004 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-09-30T01:53:20.004 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:53:20.004 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:53:20.004 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:53:20.127 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados2 2023-09-30T01:53:20.303 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:53:20.304 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-09-30T01:53:20.304 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:53:20.413 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rgw 2023-09-30T01:53:20.519 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:53:20.809 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:53:21.478 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:37 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:21.659 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:37 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:22.875 INFO:teuthology.orchestra.run.smithi007.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:53:23.285 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:53:23.366 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:53:23.368 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-09-30T01:53:23.368 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:53:23.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librbd1 2023-09-30T01:53:23.785 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:53:23.786 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-09-30T01:53:23.786 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:53:23.877 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:53:23.900 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-cephfs 2023-09-30T01:53:24.294 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:53:24.877 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:40 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:25.177 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:41 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:26.316 INFO:teuthology.orchestra.run.smithi007.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:53:26.642 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:53:26.811 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:53:26.812 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-09-30T01:53:26.812 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:53:26.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rados 2023-09-30T01:53:27.136 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:53:27.137 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-09-30T01:53:27.137 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:53:27.233 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rbd 2023-09-30T01:53:27.310 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:53:27.626 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:53:28.274 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:44 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:28.437 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:44 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:29.807 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:53:29.858 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:53:30.303 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:53:30.304 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-09-30T01:53:30.304 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:53:30.353 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:53:30.354 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2023-09-30T01:53:30.354 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:53:30.418 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rgw 2023-09-30T01:53:30.465 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-fuse 2023-09-30T01:53:30.813 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:53:30.859 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:53:31.701 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:47 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:31.763 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:47 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:33.206 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:53:33.601 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:53:33.601 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:53:33.601 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2023-09-30T01:53:33.602 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:53:33.602 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-09-30T01:53:33.602 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-30T01:53:33.602 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:53:33.602 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-09-30T01:53:33.602 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:53:33.602 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-09-30T01:53:33.602 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:53:33.603 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 90 k 2023-09-30T01:53:33.603 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 264 k 2023-09-30T01:53:33.603 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-09-30T01:53:33.710 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:53:33.712 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-09-30T01:53:33.712 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:53:33.814 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-cephfs 2023-09-30T01:53:34.136 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 169 kB/s | 90 kB 00:00 2023-09-30T01:53:34.137 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:53:34.137 INFO:teuthology.orchestra.run.smithi050.stdout:Total 169 kB/s | 90 kB 00:00 2023-09-30T01:53:34.137 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-09-30T01:53:34.150 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-09-30T01:53:34.150 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-09-30T01:53:34.206 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:53:34.210 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-09-30T01:53:34.215 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-09-30T01:53:34.328 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-09-30T01:53:34.514 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:34.826 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:35.380 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:51 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:35.870 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:35.871 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:51 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:36.822 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:53:37.320 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:53:37.321 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-09-30T01:53:37.321 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:53:37.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rbd 2023-09-30T01:53:37.829 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:53:38.060 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-09-30T01:53:38.165 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:53:38.166 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-09-30T01:53:38.166 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:53:38.166 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:53:38.166 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:53:38.291 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-mirror 2023-09-30T01:53:38.688 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:53:38.868 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:54 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:39.600 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:55 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:40.263 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:53:40.753 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:53:40.754 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-09-30T01:53:40.754 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:53:40.867 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-fuse 2023-09-30T01:53:41.262 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:53:41.654 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:53:41.655 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:53:41.655 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2023-09-30T01:53:41.655 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:53:41.655 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-09-30T01:53:41.655 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T01:53:41.655 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:53:41.655 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-09-30T01:53:41.656 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:53:41.656 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-09-30T01:53:41.656 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:53:41.656 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 3.5 M 2023-09-30T01:53:41.656 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 16 M 2023-09-30T01:53:41.656 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-09-30T01:53:43.040 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:59 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:43.121 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 2.4 MB/s | 3.5 MB 00:01 2023-09-30T01:53:43.123 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:53:43.123 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.4 MB/s | 3.5 MB 00:01 2023-09-30T01:53:43.123 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-09-30T01:53:43.140 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-09-30T01:53:43.140 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-09-30T01:53:43.218 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-09-30T01:53:43.223 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-09-30T01:53:43.714 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-09-30T01:53:43.922 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:44.991 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:53:44.991 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:53:44.992 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2023-09-30T01:53:44.992 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:53:44.992 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-09-30T01:53:44.992 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-30T01:53:44.992 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:53:44.992 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-09-30T01:53:44.992 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:53:44.992 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-09-30T01:53:44.992 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:53:44.993 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 90 k 2023-09-30T01:53:44.993 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 264 k 2023-09-30T01:53:44.993 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-09-30T01:53:45.356 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 249 kB/s | 90 kB 00:00 2023-09-30T01:53:45.356 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:53:45.357 INFO:teuthology.orchestra.run.smithi007.stdout:Total 248 kB/s | 90 kB 00:00 2023-09-30T01:53:45.357 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-09-30T01:53:45.370 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-09-30T01:53:45.371 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-09-30T01:53:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:45.386 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:01 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:45.431 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-09-30T01:53:45.436 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-09-30T01:53:45.575 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-09-30T01:53:45.830 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:46.124 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:47.161 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:47.162 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:03 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:47.602 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-09-30T01:53:47.711 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:53:47.711 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-09-30T01:53:47.711 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:53:47.711 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:53:47.711 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:53:47.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-nbd 2023-09-30T01:53:48.230 INFO:teuthology.orchestra.run.smithi050.stdout:Updating Subscription Management repositories. 2023-09-30T01:53:49.054 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:05 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:49.359 INFO:teuthology.orchestra.run.smithi007.stdout:Installed products updated. 2023-09-30T01:53:49.467 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:53:49.467 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-09-30T01:53:49.467 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:53:49.467 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:53:49.468 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:53:49.597 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-mirror 2023-09-30T01:53:49.994 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:53:50.955 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:06 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:51.006 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2023-09-30T01:53:51.007 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:53:51.007 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2023-09-30T01:53:51.007 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:53:51.007 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2023-09-30T01:53:51.007 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-30T01:53:51.007 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:53:51.007 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2023-09-30T01:53:51.007 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2023-09-30T01:53:51.008 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2023-09-30T01:53:51.008 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:53:51.008 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 173 k 2023-09-30T01:53:51.008 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 552 k 2023-09-30T01:53:51.008 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2023-09-30T01:53:51.565 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 311 kB/s | 173 kB 00:00 2023-09-30T01:53:51.565 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:53:51.566 INFO:teuthology.orchestra.run.smithi050.stdout:Total 310 kB/s | 173 kB 00:00 2023-09-30T01:53:51.566 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2023-09-30T01:53:51.581 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2023-09-30T01:53:51.581 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2023-09-30T01:53:51.643 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2023-09-30T01:53:51.648 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2023-09-30T01:53:51.779 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2023-09-30T01:53:51.970 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:52.320 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:52.903 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:53:52.904 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:53:52.904 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-09-30T01:53:52.904 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:53:52.904 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-09-30T01:53:52.904 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T01:53:52.904 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:53:52.904 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-09-30T01:53:52.904 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:53:52.904 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-09-30T01:53:52.905 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:53:52.905 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 3.5 M 2023-09-30T01:53:52.905 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 16 M 2023-09-30T01:53:52.905 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-09-30T01:53:53.354 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:53.354 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:09 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:53:54.612 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 2.0 MB/s | 3.5 MB 00:01 2023-09-30T01:53:54.613 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:53:54.614 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.0 MB/s | 3.5 MB 00:01 2023-09-30T01:53:54.614 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-09-30T01:53:54.630 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-09-30T01:53:54.631 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-09-30T01:53:54.708 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-09-30T01:53:54.712 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-09-30T01:53:55.228 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-09-30T01:53:55.429 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:55.591 INFO:teuthology.orchestra.run.smithi050.stdout:Installed products updated. 2023-09-30T01:53:55.698 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:53:55.698 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2023-09-30T01:53:55.698 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:53:55.699 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:53:55.699 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2023-09-30T01:53:55.839 DEBUG:teuthology.parallel:result is None 2023-09-30T01:53:55.868 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:56.979 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:56.979 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:12 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:54:01.047 INFO:teuthology.orchestra.run.smithi007.stdout:Installed products updated. 2023-09-30T01:54:01.153 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:01.153 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-09-30T01:54:01.153 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:54:01.153 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:01.154 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:54:01.295 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-nbd 2023-09-30T01:54:01.697 INFO:teuthology.orchestra.run.smithi007.stdout:Updating Subscription Management repositories. 2023-09-30T01:54:02.714 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:18 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:54:04.677 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-09-30T01:54:04.678 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:54:04.678 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2023-09-30T01:54:04.678 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:54:04.678 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-09-30T01:54:04.678 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-30T01:54:04.678 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:04.678 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-09-30T01:54:04.679 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-09-30T01:54:04.679 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-09-30T01:54:04.679 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:04.679 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 173 k 2023-09-30T01:54:04.679 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 552 k 2023-09-30T01:54:04.679 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-09-30T01:54:04.898 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 792 kB/s | 173 kB 00:00 2023-09-30T01:54:04.899 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:54:04.899 INFO:teuthology.orchestra.run.smithi007.stdout:Total 786 kB/s | 173 kB 00:00 2023-09-30T01:54:04.899 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-09-30T01:54:04.914 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-09-30T01:54:04.914 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-09-30T01:54:04.974 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-09-30T01:54:04.979 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-09-30T01:54:05.106 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-09-30T01:54:05.310 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:54:05.599 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:54:06.664 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:54:06.664 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:22 ago on Sat 30 Sep 2023 01:49:44 AM UTC. 2023-09-30T01:54:08.796 INFO:teuthology.orchestra.run.smithi007.stdout:Installed products updated. 2023-09-30T01:54:08.903 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:08.904 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-09-30T01:54:08.904 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:54:08.904 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:08.904 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-09-30T01:54:09.025 DEBUG:teuthology.parallel:result is None 2023-09-30T01:54:09.026 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:54:09.263 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-30T01:54:09.283 INFO:teuthology.orchestra.run.smithi007.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-30T01:54:09.283 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-30T01:54:09.283 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-30T01:54:09.284 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:54:09.434 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-30T01:54:09.458 INFO:teuthology.orchestra.run.smithi050.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-30T01:54:09.459 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-30T01:54:09.459 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-30T01:54:09.459 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-30T01:54:09.459 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-09-30T01:54:09.459 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T01:54:09.486 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-09-30T01:54:09.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T01:54:09.529 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-30T01:54:09.530 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-09-30T01:54:09.530 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T01:54:09.554 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T01:54:09.619 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-09-30T01:54:09.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T01:54:09.648 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T01:54:09.718 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-30T01:54:09.719 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-09-30T01:54:09.719 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T01:54:09.743 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T01:54:09.809 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-09-30T01:54:09.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T01:54:09.836 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T01:54:09.903 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-30T01:54:09.903 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-09-30T01:54:09.904 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T01:54:09.928 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T01:54:09.993 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-09-30T01:54:09.993 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T01:54:10.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T01:54:10.086 INFO:teuthology.run_tasks:Running task ceph... 2023-09-30T01:54:10.152 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-30T01:54:10.152 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /var/log/ceph 2023-09-30T01:54:10.155 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 777 /var/log/ceph 2023-09-30T01:54:10.180 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-30T01:54:10.180 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T01:54:10.220 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T01:54:10.246 INFO:tasks.ceph:Creating extra log directories... 2023-09-30T01:54:10.246 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T01:54:10.285 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T01:54:10.322 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-30T01:54:10.322 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-30T01:54:10.322 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407090', 'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/one workloads/rados_mon_workunits}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7407090', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', 'but it is still running', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(AUTH_BAD_CAPS\\)'], 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 9810, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'targets': {'smithi007.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL4Ne8rnE0fKgnLxndlIRGawJ6kI6jVZPJd3ia5X2z5siGW7NsynibbkY3Sa/LPpAgVwpTy1xB+DLeEy9osBQQA=', 'smithi050.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPKVvvKVeE2oRIix/AIkKakF3O4QIvkSEqHCqfZMbbIAoWcstU9WTGbXpT9G0dSp1ejGKy6wHwNz/gMXjOyAK9U='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 20, 'thrash_delay': 1}}, {'workunit': {'clients': {'client.0': ['mon/pool_ops.sh', 'mon/crush_ops.sh', 'mon/osd.sh', 'mon/caps.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-29_19:38:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-30T01:54:10.323 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T01:54:10.350 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T01:54:10.368 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T01:54:10.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T01:54:10.433 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-09-30T01:54:10.433 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T01:54:10.483 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T01:54:10.483 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2023-09-30T01:54:10.538 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T01:54:10.539 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:54:10.539 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 6h/6d Inode: 216537 Links: 1 2023-09-30T01:54:10.539 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:54:10.539 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T01:54:10.539 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-09-30 01:53:55.626524020 +0000 2023-09-30T01:54:10.539 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-09-30 01:48:58.436937290 +0000 2023-09-30T01:54:10.539 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-09-30 01:48:58.436937290 +0000 2023-09-30T01:54:10.539 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2023-09-30T01:54:10.540 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T01:54:10.602 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-09-30T01:54:10.602 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-09-30T01:54:10.602 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.00011902 s, 4.3 MB/s 2023-09-30T01:54:10.603 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T01:54:10.660 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2023-09-30T01:54:10.715 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T01:54:10.715 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:54:10.715 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 6h/6d Inode: 217860 Links: 1 2023-09-30T01:54:10.715 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:54:10.716 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T01:54:10.716 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-09-30 01:53:55.626524020 +0000 2023-09-30T01:54:10.716 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-09-30 01:48:58.985929315 +0000 2023-09-30T01:54:10.716 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-09-30 01:48:58.985929315 +0000 2023-09-30T01:54:10.716 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2023-09-30T01:54:10.716 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T01:54:10.779 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-09-30T01:54:10.779 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-09-30T01:54:10.780 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000107504 s, 4.8 MB/s 2023-09-30T01:54:10.780 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T01:54:10.837 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2023-09-30T01:54:10.892 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T01:54:10.892 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:54:10.892 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 6h/6d Inode: 217943 Links: 1 2023-09-30T01:54:10.892 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:54:10.892 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T01:54:10.892 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-09-30 01:53:55.626524020 +0000 2023-09-30T01:54:10.892 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-09-30 01:48:59.527921441 +0000 2023-09-30T01:54:10.892 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-09-30 01:48:59.527921441 +0000 2023-09-30T01:54:10.893 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2023-09-30T01:54:10.893 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T01:54:10.955 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-09-30T01:54:10.955 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-09-30T01:54:10.955 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000111533 s, 4.6 MB/s 2023-09-30T01:54:10.956 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T01:54:11.013 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2023-09-30T01:54:11.068 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T01:54:11.068 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:54:11.068 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 6h/6d Inode: 218022 Links: 1 2023-09-30T01:54:11.068 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:54:11.068 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T01:54:11.068 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-09-30 01:53:55.626524020 +0000 2023-09-30T01:54:11.068 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-09-30 01:49:00.077913451 +0000 2023-09-30T01:54:11.069 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-09-30 01:49:00.077913451 +0000 2023-09-30T01:54:11.069 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2023-09-30T01:54:11.069 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T01:54:11.131 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-09-30T01:54:11.132 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-09-30T01:54:11.132 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000103951 s, 4.9 MB/s 2023-09-30T01:54:11.133 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T01:54:11.189 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-30T01:54:11.189 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-09-30T01:54:11.189 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T01:54:11.206 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T01:54:11.206 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_1 2023-09-30T01:54:11.263 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T01:54:11.263 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:54:11.264 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 6h/6d Inode: 217709 Links: 1 2023-09-30T01:54:11.264 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:54:11.264 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T01:54:11.264 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2023-09-30 01:53:44.113634775 +0000 2023-09-30T01:54:11.264 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2023-09-30 01:48:58.472760841 +0000 2023-09-30T01:54:11.264 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2023-09-30 01:48:58.472760841 +0000 2023-09-30T01:54:11.264 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2023-09-30T01:54:11.264 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T01:54:11.329 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2023-09-30T01:54:11.330 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2023-09-30T01:54:11.330 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000117792 s, 4.3 MB/s 2023-09-30T01:54:11.331 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T01:54:11.388 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_2 2023-09-30T01:54:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T01:54:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:54:11.446 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 6h/6d Inode: 219021 Links: 1 2023-09-30T01:54:11.446 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:54:11.446 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T01:54:11.446 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2023-09-30 01:53:44.113634775 +0000 2023-09-30T01:54:11.446 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2023-09-30 01:48:59.006747775 +0000 2023-09-30T01:54:11.446 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2023-09-30 01:48:59.006747775 +0000 2023-09-30T01:54:11.446 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2023-09-30T01:54:11.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T01:54:11.511 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2023-09-30T01:54:11.511 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2023-09-30T01:54:11.511 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000102304 s, 5.0 MB/s 2023-09-30T01:54:11.512 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T01:54:11.569 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_3 2023-09-30T01:54:11.625 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T01:54:11.625 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:54:11.626 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 6h/6d Inode: 211793 Links: 1 2023-09-30T01:54:11.626 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:54:11.626 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T01:54:11.626 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2023-09-30 01:53:44.113634775 +0000 2023-09-30T01:54:11.626 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2023-09-30 01:48:59.548734513 +0000 2023-09-30T01:54:11.626 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2023-09-30 01:48:59.548734513 +0000 2023-09-30T01:54:11.626 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2023-09-30T01:54:11.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T01:54:11.690 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2023-09-30T01:54:11.690 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2023-09-30T01:54:11.690 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000118345 s, 4.3 MB/s 2023-09-30T01:54:11.691 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T01:54:11.748 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_4 2023-09-30T01:54:11.804 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T01:54:11.805 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T01:54:11.805 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 6h/6d Inode: 220337 Links: 1 2023-09-30T01:54:11.805 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T01:54:11.805 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T01:54:11.805 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2023-09-30 01:53:44.113634775 +0000 2023-09-30T01:54:11.805 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2023-09-30 01:49:00.082721448 +0000 2023-09-30T01:54:11.805 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2023-09-30 01:49:00.082721448 +0000 2023-09-30T01:54:11.805 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2023-09-30T01:54:11.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T01:54:11.869 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2023-09-30T01:54:11.869 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2023-09-30T01:54:11.870 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.00011957 s, 4.3 MB/s 2023-09-30T01:54:11.870 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T01:54:11.927 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-09-30T01:54:11.927 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2023-09-30T01:54:11.927 INFO:tasks.ceph:Generating config... 2023-09-30T01:54:11.929 INFO:tasks.ceph:[client] debug monc = 20 2023-09-30T01:54:11.929 INFO:tasks.ceph:[client] debug ms = 1 2023-09-30T01:54:11.929 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-09-30T01:54:11.929 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-09-30T01:54:11.929 INFO:tasks.ceph:[global] ms inject delay max = 1 2023-09-30T01:54:11.929 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2023-09-30T01:54:11.929 INFO:tasks.ceph:[global] ms inject delay type = mon 2023-09-30T01:54:11.929 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2023-09-30T01:54:11.929 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2023-09-30T01:54:11.929 INFO:tasks.ceph:[global] ms type = async 2023-09-30T01:54:11.929 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-30T01:54:11.930 INFO:tasks.ceph:[mgr] debug monc = 10 2023-09-30T01:54:11.930 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-30T01:54:11.930 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-30T01:54:11.930 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-30T01:54:11.930 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-30T01:54:11.930 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2023-09-30T01:54:11.930 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-09-30T01:54:11.930 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-09-30T01:54:11.930 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-09-30T01:54:11.930 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2023-09-30T01:54:11.930 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2023-09-30T01:54:11.931 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-09-30T01:54:11.931 INFO:tasks.ceph:[mon] paxos service trim min = 5 2023-09-30T01:54:11.931 INFO:tasks.ceph:[osd] bdev async discard = True 2023-09-30T01:54:11.931 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-09-30T01:54:11.931 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-09-30T01:54:11.931 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-09-30T01:54:11.931 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-09-30T01:54:11.931 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-09-30T01:54:11.932 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-09-30T01:54:11.932 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-30T01:54:11.932 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-30T01:54:11.932 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-09-30T01:54:11.932 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-09-30T01:54:11.932 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-09-30T01:54:11.932 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-09-30T01:54:11.932 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-09-30T01:54:11.932 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-09-30T01:54:11.932 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-09-30T01:54:11.932 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-09-30T01:54:11.932 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-09-30T01:54:11.933 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2023-09-30T01:54:11.933 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-09-30T01:54:11.933 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-09-30T01:54:11.933 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-09-30T01:54:11.933 INFO:tasks.ceph:Setting up mon.a... 2023-09-30T01:54:11.933 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-30T01:54:11.989 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/ceph.keyring 2023-09-30T01:54:11.991 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-30T01:54:12.032 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T01:54:12.098 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.7'), ('mon.c', '[v2:172.21.15.7:3301,v1:172.21.15.7:6790]'), ('mon.b', '172.21.15.50')] 2023-09-30T01:54:12.099 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.7,[v2:172.21.15.7:3301,v1:172.21.15.7:6790],172.21.15.50', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-09-30T01:54:12.100 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-09-30T01:54:12.100 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T01:54:12.154 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.7 --addv c '[v2:172.21.15.7:3301,v1:172.21.15.7:6790]' --add b 172.21.15.50 --print /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:54:12.236 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:54:12.236 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: generated fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:12.237 INFO:teuthology.orchestra.run.smithi007.stdout:setting min_mon_release = octopus 2023-09-30T01:54:12.237 INFO:teuthology.orchestra.run.smithi007.stdout:epoch 0 2023-09-30T01:54:12.237 INFO:teuthology.orchestra.run.smithi007.stdout:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:12.237 INFO:teuthology.orchestra.run.smithi007.stdout:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:12.237 INFO:teuthology.orchestra.run.smithi007.stdout:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:12.237 INFO:teuthology.orchestra.run.smithi007.stdout:min_mon_release 15 (octopus) 2023-09-30T01:54:12.237 INFO:teuthology.orchestra.run.smithi007.stdout:election_strategy: 1 2023-09-30T01:54:12.237 INFO:teuthology.orchestra.run.smithi007.stdout:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:12.237 INFO:teuthology.orchestra.run.smithi007.stdout:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:12.237 INFO:teuthology.orchestra.run.smithi007.stdout:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:12.238 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-09-30T01:54:12.239 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T01:54:12.294 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2ea1044d-d984-49b9-baad-7f36bd454a19... 2023-09-30T01:54:12.295 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T01:54:12.336 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T01:54:12.378 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2023-09-30T01:54:12.378 INFO:teuthology.orchestra.run.smithi007.stdout: chdir = "" 2023-09-30T01:54:12.378 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T01:54:12.379 INFO:teuthology.orchestra.run.smithi007.stdout: auth supported = cephx 2023-09-30T01:54:12.379 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.379 INFO:teuthology.orchestra.run.smithi007.stdout: filestore xattr use omap = true 2023-09-30T01:54:12.379 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.379 INFO:teuthology.orchestra.run.smithi007.stdout: mon clock drift allowed = 1.000 2023-09-30T01:54:12.379 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.379 INFO:teuthology.orchestra.run.smithi007.stdout: osd crush chooseleaf type = 0 2023-09-30T01:54:12.379 INFO:teuthology.orchestra.run.smithi007.stdout: auth debug = true 2023-09-30T01:54:12.379 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.379 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on old message = true 2023-09-30T01:54:12.379 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on bug = true 2023-09-30T01:54:12.379 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.380 INFO:teuthology.orchestra.run.smithi007.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T01:54:12.380 INFO:teuthology.orchestra.run.smithi007.stdout: mon pg warn max object skew = 0 2023-09-30T01:54:12.380 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.380 INFO:teuthology.orchestra.run.smithi007.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T01:54:12.380 INFO:teuthology.orchestra.run.smithi007.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T01:54:12.380 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.380 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default size = 2 2023-09-30T01:54:12.380 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.380 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow primary affinity = true 2023-09-30T01:54:12.380 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow pg remap = true 2023-09-30T01:54:12.380 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on legacy crush tunables = false 2023-09-30T01:54:12.381 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on crush straw calc version zero = false 2023-09-30T01:54:12.381 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on no sortbitwise = false 2023-09-30T01:54:12.381 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on osd down out interval zero = false 2023-09-30T01:54:12.381 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on too few osds = false 2023-09-30T01:54:12.381 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T01:54:12.381 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T01:54:12.381 INFO:teuthology.orchestra.run.smithi007.stdout: mon_allow_pool_size_one = true 2023-09-30T01:54:12.381 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.381 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T01:54:12.381 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.381 INFO:teuthology.orchestra.run.smithi007.stdout: osd default data pool replay window = 5 2023-09-30T01:54:12.381 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.382 INFO:teuthology.orchestra.run.smithi007.stdout: mon allow pool delete = true 2023-09-30T01:54:12.382 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.382 INFO:teuthology.orchestra.run.smithi007.stdout: mon cluster log file level = debug 2023-09-30T01:54:12.382 INFO:teuthology.orchestra.run.smithi007.stdout: debug asserts on shutdown = true 2023-09-30T01:54:12.382 INFO:teuthology.orchestra.run.smithi007.stdout: mon health detail to clog = false 2023-09-30T01:54:12.382 INFO:teuthology.orchestra.run.smithi007.stdout: mon host = "172.21.15.7,[v2:172.21.15.7:3301,v1:172.21.15.7:6790],172.21.15.50" 2023-09-30T01:54:12.382 INFO:teuthology.orchestra.run.smithi007.stdout: mon client directed command retry = 5 2023-09-30T01:54:12.382 INFO:teuthology.orchestra.run.smithi007.stdout: mon election default strategy = 1 2023-09-30T01:54:12.382 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject delay max = 1 2023-09-30T01:54:12.382 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject delay probability = 0.005 2023-09-30T01:54:12.382 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject delay type = mon 2023-09-30T01:54:12.382 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject internal delays = 0.002 2023-09-30T01:54:12.383 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject socket failures = 2500 2023-09-30T01:54:12.383 INFO:teuthology.orchestra.run.smithi007.stdout: ms type = async 2023-09-30T01:54:12.383 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:12.383 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.383 INFO:teuthology.orchestra.run.smithi007.stdout:[osd] 2023-09-30T01:54:12.383 INFO:teuthology.orchestra.run.smithi007.stdout: osd journal size = 100 2023-09-30T01:54:12.385 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.385 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub load threshold = 5.0 2023-09-30T01:54:12.385 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub max interval = 600 2023-09-30T01:54:12.385 INFO:teuthology.orchestra.run.smithi007.stdout: osd mclock profile = high_recovery_ops 2023-09-30T01:54:12.385 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.385 INFO:teuthology.orchestra.run.smithi007.stdout: osd recover clone overlap = true 2023-09-30T01:54:12.385 INFO:teuthology.orchestra.run.smithi007.stdout: osd recovery max chunk = 1048576 2023-09-30T01:54:12.385 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.385 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug shutdown = true 2023-09-30T01:54:12.386 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug op order = true 2023-09-30T01:54:12.386 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify stray on activate = true 2023-09-30T01:54:12.386 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.386 INFO:teuthology.orchestra.run.smithi007.stdout: osd open classes on start = true 2023-09-30T01:54:12.386 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug pg log writeout = true 2023-09-30T01:54:12.386 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.386 INFO:teuthology.orchestra.run.smithi007.stdout: osd deep scrub update digest min age = 30 2023-09-30T01:54:12.386 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.386 INFO:teuthology.orchestra.run.smithi007.stdout: osd map max advance = 10 2023-09-30T01:54:12.386 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.386 INFO:teuthology.orchestra.run.smithi007.stdout: journal zero on create = true 2023-09-30T01:54:12.386 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.387 INFO:teuthology.orchestra.run.smithi007.stdout: filestore ondisk finisher threads = 3 2023-09-30T01:54:12.387 INFO:teuthology.orchestra.run.smithi007.stdout: filestore apply finisher threads = 3 2023-09-30T01:54:12.387 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.387 INFO:teuthology.orchestra.run.smithi007.stdout: bdev debug aio = true 2023-09-30T01:54:12.387 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug misdirected ops = true 2023-09-30T01:54:12.387 INFO:teuthology.orchestra.run.smithi007.stdout: bdev async discard = True 2023-09-30T01:54:12.387 INFO:teuthology.orchestra.run.smithi007.stdout: bdev enable discard = True 2023-09-30T01:54:12.387 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore block size = 96636764160 2023-09-30T01:54:12.387 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore fsck on mount = True 2023-09-30T01:54:12.387 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore zero block detection = True 2023-09-30T01:54:12.387 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluefs = 20 2023-09-30T01:54:12.387 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluestore = 20 2023-09-30T01:54:12.388 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2023-09-30T01:54:12.388 INFO:teuthology.orchestra.run.smithi007.stdout: debug osd = 20 2023-09-30T01:54:12.388 INFO:teuthology.orchestra.run.smithi007.stdout: debug rocksdb = 10 2023-09-30T01:54:12.388 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T01:54:12.388 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd full ratio = 0.9 2023-09-30T01:54:12.388 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd nearfull ratio = 0.8 2023-09-30T01:54:12.388 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify cached snaps = True 2023-09-30T01:54:12.388 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify missing on start = True 2023-09-30T01:54:12.388 INFO:teuthology.orchestra.run.smithi007.stdout: osd failsafe full ratio = 0.95 2023-09-30T01:54:12.388 INFO:teuthology.orchestra.run.smithi007.stdout: osd mclock override recovery settings = True 2023-09-30T01:54:12.388 INFO:teuthology.orchestra.run.smithi007.stdout: osd memory target = 1610612736 2023-09-30T01:54:12.388 INFO:teuthology.orchestra.run.smithi007.stdout: osd objectstore = bluestore 2023-09-30T01:54:12.389 INFO:teuthology.orchestra.run.smithi007.stdout: osd op queue = debug_random 2023-09-30T01:54:12.389 INFO:teuthology.orchestra.run.smithi007.stdout: osd op queue cut off = debug_random 2023-09-30T01:54:12.389 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.389 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr] 2023-09-30T01:54:12.389 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2023-09-30T01:54:12.389 INFO:teuthology.orchestra.run.smithi007.stdout: debug mgr = 20 2023-09-30T01:54:12.389 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2023-09-30T01:54:12.389 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2023-09-30T01:54:12.389 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2023-09-30T01:54:12.389 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2023-09-30T01:54:12.389 INFO:teuthology.orchestra.run.smithi007.stdout: mgr/telemetry/nag = false 2023-09-30T01:54:12.389 INFO:teuthology.orchestra.run.smithi007.stdout: debug monc = 10 2023-09-30T01:54:12.390 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.390 INFO:teuthology.orchestra.run.smithi007.stdout:[mon] 2023-09-30T01:54:12.390 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2023-09-30T01:54:12.390 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2023-09-30T01:54:12.390 INFO:teuthology.orchestra.run.smithi007.stdout: debug paxos = 20 2023-09-30T01:54:12.390 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2023-09-30T01:54:12.390 INFO:teuthology.orchestra.run.smithi007.stdout: mon data avail warn = 5 2023-09-30T01:54:12.390 INFO:teuthology.orchestra.run.smithi007.stdout: mon mgr mkfs grace = 240 2023-09-30T01:54:12.390 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2023-09-30T01:54:12.390 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd reporter subtree level = osd 2023-09-30T01:54:12.390 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd prime pg temp = true 2023-09-30T01:54:12.391 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2023-09-30T01:54:12.391 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.391 INFO:teuthology.orchestra.run.smithi007.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T01:54:12.391 INFO:teuthology.orchestra.run.smithi007.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T01:54:12.391 INFO:teuthology.orchestra.run.smithi007.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T01:54:12.391 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.391 INFO:teuthology.orchestra.run.smithi007.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T01:54:12.391 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T01:54:12.391 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T01:54:12.391 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.391 INFO:teuthology.orchestra.run.smithi007.stdout: # 1m isn't quite enough 2023-09-30T01:54:12.391 INFO:teuthology.orchestra.run.smithi007.stdout: mon_down_mkfs_grace = 2m 2023-09-30T01:54:12.392 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.392 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_filestore_osds = false 2023-09-30T01:54:12.392 INFO:teuthology.orchestra.run.smithi007.stdout: mon min osdmap epochs = 25 2023-09-30T01:54:12.392 INFO:teuthology.orchestra.run.smithi007.stdout: mon osdmap full prune interval = 2 2023-09-30T01:54:12.392 INFO:teuthology.orchestra.run.smithi007.stdout: mon osdmap full prune min = 15 2023-09-30T01:54:12.392 INFO:teuthology.orchestra.run.smithi007.stdout: mon osdmap full prune txsize = 2 2023-09-30T01:54:12.392 INFO:teuthology.orchestra.run.smithi007.stdout: mon scrub inject crc mismatch = 0.01 2023-09-30T01:54:12.392 INFO:teuthology.orchestra.run.smithi007.stdout: mon scrub inject missing keys = 0.05 2023-09-30T01:54:12.392 INFO:teuthology.orchestra.run.smithi007.stdout: mon scrub interval = 300 2023-09-30T01:54:12.392 INFO:teuthology.orchestra.run.smithi007.stdout: paxos service trim min = 5 2023-09-30T01:54:12.392 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:12.392 INFO:teuthology.orchestra.run.smithi007.stdout:[client] 2023-09-30T01:54:12.393 INFO:teuthology.orchestra.run.smithi007.stdout: rgw cache enabled = true 2023-09-30T01:54:12.393 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable ops log = true 2023-09-30T01:54:12.393 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable usage log = true 2023-09-30T01:54:12.393 INFO:teuthology.orchestra.run.smithi007.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T01:54:12.393 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T01:54:12.393 INFO:teuthology.orchestra.run.smithi007.stdout: debug monc = 20 2023-09-30T01:54:12.393 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2023-09-30T01:54:12.393 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.a] 2023-09-30T01:54:12.393 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.c] 2023-09-30T01:54:12.393 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.b] 2023-09-30T01:54:12.395 INFO:teuthology.orchestra.run.smithi050.stdout:[global] 2023-09-30T01:54:12.395 INFO:teuthology.orchestra.run.smithi050.stdout: chdir = "" 2023-09-30T01:54:12.395 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T01:54:12.395 INFO:teuthology.orchestra.run.smithi050.stdout: auth supported = cephx 2023-09-30T01:54:12.395 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.395 INFO:teuthology.orchestra.run.smithi050.stdout: filestore xattr use omap = true 2023-09-30T01:54:12.395 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.395 INFO:teuthology.orchestra.run.smithi050.stdout: mon clock drift allowed = 1.000 2023-09-30T01:54:12.395 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: osd crush chooseleaf type = 0 2023-09-30T01:54:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: auth debug = true 2023-09-30T01:54:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on old message = true 2023-09-30T01:54:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on bug = true 2023-09-30T01:54:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T01:54:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: mon pg warn max object skew = 0 2023-09-30T01:54:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T01:54:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T01:54:12.396 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default size = 2 2023-09-30T01:54:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow primary affinity = true 2023-09-30T01:54:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow pg remap = true 2023-09-30T01:54:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on legacy crush tunables = false 2023-09-30T01:54:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on crush straw calc version zero = false 2023-09-30T01:54:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on no sortbitwise = false 2023-09-30T01:54:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on osd down out interval zero = false 2023-09-30T01:54:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on too few osds = false 2023-09-30T01:54:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T01:54:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T01:54:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: mon_allow_pool_size_one = true 2023-09-30T01:54:12.397 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.398 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T01:54:12.398 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.398 INFO:teuthology.orchestra.run.smithi050.stdout: osd default data pool replay window = 5 2023-09-30T01:54:12.398 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.398 INFO:teuthology.orchestra.run.smithi050.stdout: mon allow pool delete = true 2023-09-30T01:54:12.398 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: mon cluster log file level = debug 2023-09-30T01:54:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: debug asserts on shutdown = true 2023-09-30T01:54:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: mon health detail to clog = false 2023-09-30T01:54:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: mon host = "172.21.15.7,[v2:172.21.15.7:3301,v1:172.21.15.7:6790],172.21.15.50" 2023-09-30T01:54:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: mon client directed command retry = 5 2023-09-30T01:54:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: mon election default strategy = 1 2023-09-30T01:54:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: ms inject delay max = 1 2023-09-30T01:54:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: ms inject delay probability = 0.005 2023-09-30T01:54:12.399 INFO:teuthology.orchestra.run.smithi050.stdout: ms inject delay type = mon 2023-09-30T01:54:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: ms inject internal delays = 0.002 2023-09-30T01:54:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: ms inject socket failures = 2500 2023-09-30T01:54:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: ms type = async 2023-09-30T01:54:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: fsid = 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.400 INFO:teuthology.orchestra.run.smithi050.stdout:[osd] 2023-09-30T01:54:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: osd journal size = 100 2023-09-30T01:54:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub load threshold = 5.0 2023-09-30T01:54:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub max interval = 600 2023-09-30T01:54:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: osd mclock profile = high_recovery_ops 2023-09-30T01:54:12.400 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: osd recover clone overlap = true 2023-09-30T01:54:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: osd recovery max chunk = 1048576 2023-09-30T01:54:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug shutdown = true 2023-09-30T01:54:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug op order = true 2023-09-30T01:54:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify stray on activate = true 2023-09-30T01:54:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: osd open classes on start = true 2023-09-30T01:54:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug pg log writeout = true 2023-09-30T01:54:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: osd deep scrub update digest min age = 30 2023-09-30T01:54:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: osd map max advance = 10 2023-09-30T01:54:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: journal zero on create = true 2023-09-30T01:54:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: filestore ondisk finisher threads = 3 2023-09-30T01:54:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: filestore apply finisher threads = 3 2023-09-30T01:54:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: bdev debug aio = true 2023-09-30T01:54:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug misdirected ops = true 2023-09-30T01:54:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: bdev async discard = True 2023-09-30T01:54:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: bdev enable discard = True 2023-09-30T01:54:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore block size = 96636764160 2023-09-30T01:54:12.402 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore fsck on mount = True 2023-09-30T01:54:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore zero block detection = True 2023-09-30T01:54:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluefs = 20 2023-09-30T01:54:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluestore = 20 2023-09-30T01:54:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-09-30T01:54:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: debug osd = 20 2023-09-30T01:54:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: debug rocksdb = 10 2023-09-30T01:54:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T01:54:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd full ratio = 0.9 2023-09-30T01:54:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd nearfull ratio = 0.8 2023-09-30T01:54:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify cached snaps = True 2023-09-30T01:54:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify missing on start = True 2023-09-30T01:54:12.403 INFO:teuthology.orchestra.run.smithi050.stdout: osd failsafe full ratio = 0.95 2023-09-30T01:54:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: osd mclock override recovery settings = True 2023-09-30T01:54:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: osd memory target = 1610612736 2023-09-30T01:54:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: osd objectstore = bluestore 2023-09-30T01:54:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: osd op queue = debug_random 2023-09-30T01:54:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: osd op queue cut off = debug_random 2023-09-30T01:54:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.404 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr] 2023-09-30T01:54:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-09-30T01:54:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: debug mgr = 20 2023-09-30T01:54:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2023-09-30T01:54:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2023-09-30T01:54:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2023-09-30T01:54:12.404 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2023-09-30T01:54:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: mgr/telemetry/nag = false 2023-09-30T01:54:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: debug monc = 10 2023-09-30T01:54:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.405 INFO:teuthology.orchestra.run.smithi050.stdout:[mon] 2023-09-30T01:54:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-09-30T01:54:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2023-09-30T01:54:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: debug paxos = 20 2023-09-30T01:54:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2023-09-30T01:54:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: mon data avail warn = 5 2023-09-30T01:54:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: mon mgr mkfs grace = 240 2023-09-30T01:54:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2023-09-30T01:54:12.405 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd reporter subtree level = osd 2023-09-30T01:54:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd prime pg temp = true 2023-09-30T01:54:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2023-09-30T01:54:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T01:54:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T01:54:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T01:54:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T01:54:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T01:54:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T01:54:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: # 1m isn't quite enough 2023-09-30T01:54:12.406 INFO:teuthology.orchestra.run.smithi050.stdout: mon_down_mkfs_grace = 2m 2023-09-30T01:54:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_filestore_osds = false 2023-09-30T01:54:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: mon min osdmap epochs = 25 2023-09-30T01:54:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: mon osdmap full prune interval = 2 2023-09-30T01:54:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: mon osdmap full prune min = 15 2023-09-30T01:54:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: mon osdmap full prune txsize = 2 2023-09-30T01:54:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: mon scrub inject crc mismatch = 0.01 2023-09-30T01:54:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: mon scrub inject missing keys = 0.05 2023-09-30T01:54:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: mon scrub interval = 300 2023-09-30T01:54:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: paxos service trim min = 5 2023-09-30T01:54:12.407 INFO:teuthology.orchestra.run.smithi050.stdout: 2023-09-30T01:54:12.407 INFO:teuthology.orchestra.run.smithi050.stdout:[client] 2023-09-30T01:54:12.408 INFO:teuthology.orchestra.run.smithi050.stdout: rgw cache enabled = true 2023-09-30T01:54:12.408 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable ops log = true 2023-09-30T01:54:12.408 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable usage log = true 2023-09-30T01:54:12.408 INFO:teuthology.orchestra.run.smithi050.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T01:54:12.408 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T01:54:12.408 INFO:teuthology.orchestra.run.smithi050.stdout: debug monc = 20 2023-09-30T01:54:12.408 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2023-09-30T01:54:12.408 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.a] 2023-09-30T01:54:12.408 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.c] 2023-09-30T01:54:12.408 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.b] 2023-09-30T01:54:12.410 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-30T01:54:12.410 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-30T01:54:12.501 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-30T01:54:12.502 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-09-30T01:54:12.502 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-30T01:54:12.517 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-09-30T01:54:12.517 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-30T01:54:12.571 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi007.front.sepia.ceph.com 2023-09-30T01:54:12.572 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-09-30T01:54:12.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T01:54:12.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T01:54:12.647 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-09-30T01:54:12.648 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:54:12.703 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi050.front.sepia.ceph.com 2023-09-30T01:54:12.703 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-09-30T01:54:12.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T01:54:12.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T01:54:12.743 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-09-30T01:54:12.743 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:54:12.798 INFO:tasks.ceph:Setting up mon nodes... 2023-09-30T01:54:12.799 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-30T01:54:12.799 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T01:54:12.918 INFO:teuthology.orchestra.run.smithi050.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T01:54:12.921 INFO:tasks.ceph:Setting up mds nodes... 2023-09-30T01:54:12.921 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-30T01:54:12.921 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-30T01:54:12.961 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-30T01:54:12.974 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-30T01:54:12.974 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2023-09-30T01:54:12.974 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-30T01:54:13.000 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-30T01:54:13.000 INFO:tasks.ceph:role: osd.0 2023-09-30T01:54:13.000 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com 2023-09-30T01:54:13.000 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T01:54:13.065 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:54:13.065 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:54:13.065 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:54:13.065 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 2023-09-30T01:54:13.066 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:54:13.066 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:54:13.066 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:54:13.066 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:54:13.066 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:54:13.066 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:54:13.088 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2023-09-30T01:54:13.089 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2023-09-30T01:54:13.089 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-30T01:54:13.169 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-30T01:54:13.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-30T01:54:13.304 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-30T01:54:13.304 INFO:tasks.ceph:role: osd.1 2023-09-30T01:54:13.305 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com 2023-09-30T01:54:13.305 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T01:54:13.369 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:54:13.369 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:54:13.369 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:54:13.369 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 2023-09-30T01:54:13.370 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:54:13.370 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:54:13.370 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:54:13.370 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:54:13.370 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:54:13.370 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:54:13.391 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2023-09-30T01:54:13.393 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2023-09-30T01:54:13.393 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-09-30T01:54:13.470 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-30T01:54:13.539 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-30T01:54:13.605 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-30T01:54:13.605 INFO:tasks.ceph:role: osd.2 2023-09-30T01:54:13.605 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com 2023-09-30T01:54:13.606 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T01:54:13.670 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:54:13.671 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:54:13.671 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:54:13.671 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 2023-09-30T01:54:13.671 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:54:13.671 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:54:13.671 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:54:13.671 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:54:13.672 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:54:13.672 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:54:13.694 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2023-09-30T01:54:13.695 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2023-09-30T01:54:13.695 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-09-30T01:54:13.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-30T01:54:13.843 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:54:13.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:13.943+0000 7f705d8f1540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-09-30T01:54:13.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:13.943+0000 7f705d8f1540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-09-30T01:54:13.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:13.943+0000 7f705d8f1540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T01:54:13.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:13.943+0000 7f705d8f1540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T01:54:13.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:13.943+0000 7f705d8f1540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T01:54:13.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:13.944+0000 7f705d8f1540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-09-30T01:54:14.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:14.465+0000 7f705d8f1540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:54:15.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:15.227+0000 7f705d8f1540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:54:15.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:15.990+0000 7f705d8f1540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:54:16.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-30T01:54:16.814 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:54:16.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:16.901+0000 7fcd09a18540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-09-30T01:54:16.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:16.901+0000 7fcd09a18540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-09-30T01:54:16.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:16.901+0000 7fcd09a18540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T01:54:16.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:16.901+0000 7fcd09a18540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T01:54:16.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:16.901+0000 7fcd09a18540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T01:54:16.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:16.901+0000 7fcd09a18540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-09-30T01:54:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:17.431+0000 7fcd09a18540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:54:18.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:18.197+0000 7fcd09a18540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:54:18.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:18.963+0000 7fcd09a18540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:54:19.753 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-30T01:54:19.783 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:54:19.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:19.871+0000 7f86e048f540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-09-30T01:54:19.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:19.871+0000 7f86e048f540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-09-30T01:54:19.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:19.871+0000 7f86e048f540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T01:54:19.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:19.871+0000 7f86e048f540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T01:54:19.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:19.871+0000 7f86e048f540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T01:54:19.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:19.871+0000 7f86e048f540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-09-30T01:54:20.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:20.392+0000 7f86e048f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:54:21.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:21.155+0000 7f86e048f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:54:21.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:21.918+0000 7f86e048f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:54:22.709 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-30T01:54:22.740 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-09-30T01:54:22.773 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-09-30T01:54:22.773 INFO:tasks.ceph:role: osd.3 2023-09-30T01:54:22.773 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com 2023-09-30T01:54:22.774 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T01:54:22.843 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:54:22.843 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:54:22.844 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:54:22.844 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 2023-09-30T01:54:22.844 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:54:22.844 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:54:22.844 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:54:22.844 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:54:22.844 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:54:22.844 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:54:22.866 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2023-09-30T01:54:22.868 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2023-09-30T01:54:22.868 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2023-09-30T01:54:22.950 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-09-30T01:54:23.020 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-09-30T01:54:23.088 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-09-30T01:54:23.088 INFO:tasks.ceph:role: osd.4 2023-09-30T01:54:23.088 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com 2023-09-30T01:54:23.089 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T01:54:23.154 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:54:23.155 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:54:23.155 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:54:23.155 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 2023-09-30T01:54:23.155 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:54:23.155 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:54:23.155 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:54:23.155 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:54:23.155 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:54:23.155 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:54:23.178 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2023-09-30T01:54:23.179 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2023-09-30T01:54:23.179 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2023-09-30T01:54:23.258 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-09-30T01:54:23.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-09-30T01:54:23.392 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-09-30T01:54:23.392 INFO:tasks.ceph:role: osd.5 2023-09-30T01:54:23.392 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com 2023-09-30T01:54:23.392 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T01:54:23.457 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T01:54:23.458 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T01:54:23.458 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T01:54:23.458 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 2023-09-30T01:54:23.458 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T01:54:23.458 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2023-09-30T01:54:23.458 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T01:54:23.458 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T01:54:23.458 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T01:54:23.458 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T01:54:23.479 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2023-09-30T01:54:23.481 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2023-09-30T01:54:23.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2023-09-30T01:54:23.558 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-09-30T01:54:23.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:54:23.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:23.724+0000 7f01f9393540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-09-30T01:54:23.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:23.725+0000 7f01f9393540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-09-30T01:54:23.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:23.725+0000 7f01f9393540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T01:54:23.725 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:23.725+0000 7f01f9393540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T01:54:23.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:23.725+0000 7f01f9393540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T01:54:23.726 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:23.725+0000 7f01f9393540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-09-30T01:54:24.246 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:24.246+0000 7f01f9393540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:54:25.009 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:25.009+0000 7f01f9393540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:54:25.771 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:25.771+0000 7f01f9393540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:54:26.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-09-30T01:54:26.596 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:54:26.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:26.683+0000 7f99efc4b540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-09-30T01:54:26.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:26.683+0000 7f99efc4b540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-09-30T01:54:26.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:26.684+0000 7f99efc4b540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T01:54:26.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:26.684+0000 7f99efc4b540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T01:54:26.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:26.684+0000 7f99efc4b540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T01:54:26.684 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:26.684+0000 7f99efc4b540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-09-30T01:54:27.206 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:27.206+0000 7f99efc4b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:54:27.970 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:27.971+0000 7f99efc4b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:54:28.734 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:28.734+0000 7f99efc4b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:54:29.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-09-30T01:54:29.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:54:29.643 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:29.643+0000 7f9e67cba540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-09-30T01:54:29.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:29.643+0000 7f9e67cba540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-09-30T01:54:29.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:29.643+0000 7f9e67cba540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T01:54:29.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:29.643+0000 7f9e67cba540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T01:54:29.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:29.643+0000 7f9e67cba540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T01:54:29.644 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:29.643+0000 7f9e67cba540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-09-30T01:54:30.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:30.166+0000 7f9e67cba540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:54:30.935 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:30.935+0000 7f9e67cba540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:54:31.700 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:31.700+0000 7f9e67cba540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T01:54:32.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-09-30T01:54:32.524 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-30T01:54:32.525 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-09-30T01:54:32.525 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-30T01:54:32.556 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-09-30T01:54:32.556 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-30T01:54:32.623 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-09-30T01:54:32.624 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-30T01:54:32.691 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-09-30T01:54:32.691 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-30T01:54:32.721 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-09-30T01:54:32.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-09-30T01:54:32.792 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-09-30T01:54:32.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-09-30T01:54:32.862 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-09-30T01:54:32.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-09-30T01:54:32.929 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-09-30T01:54:32.929 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-30T01:54:32.986 INFO:tasks.ceph:Adding keys to all mons... 2023-09-30T01:54:32.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T01:54:32.988 DEBUG:teuthology.orchestra.run.smithi050:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T01:54:33.029 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.0] 2023-09-30T01:54:33.029 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDFfxdlvg5NOBAABIlPjushJmQchWdSyBXhGg== 2023-09-30T01:54:33.029 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.1] 2023-09-30T01:54:33.029 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDIfxdl/dDGNRAAuMg1oi8LO/HfZc0MnYPnCQ== 2023-09-30T01:54:33.029 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.2] 2023-09-30T01:54:33.029 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDLfxdl0bT6MxAADnZU1Vhme1dV49yD2vGKgQ== 2023-09-30T01:54:33.029 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr.x] 2023-09-30T01:54:33.029 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDEfxdl7NnRNhAAbm5KFLrl3sFGIKx38x+4+A== 2023-09-30T01:54:33.030 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.3] 2023-09-30T01:54:33.030 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDPfxdlhTVCKxAAev9OseT/PoFF0pPpWgS60Q== 2023-09-30T01:54:33.030 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.4] 2023-09-30T01:54:33.030 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDSfxdlaDXPKBAAJmFgO+bfRyzpVoW6EPjvuA== 2023-09-30T01:54:33.030 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.5] 2023-09-30T01:54:33.030 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDVfxdlRL1mJhAAfZmuwQ3tzZovV4gykf5jAw== 2023-09-30T01:54:33.030 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2023-09-30T01:54:33.030 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDEfxdl6UtYORAA9LgkXSKq1pQqY556VUy2fw== 2023-09-30T01:54:33.051 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.0] 2023-09-30T01:54:33.051 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDFfxdlvg5NOBAABIlPjushJmQchWdSyBXhGg== 2023-09-30T01:54:33.051 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.1] 2023-09-30T01:54:33.051 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDIfxdl/dDGNRAAuMg1oi8LO/HfZc0MnYPnCQ== 2023-09-30T01:54:33.051 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.2] 2023-09-30T01:54:33.051 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDLfxdl0bT6MxAADnZU1Vhme1dV49yD2vGKgQ== 2023-09-30T01:54:33.051 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr.x] 2023-09-30T01:54:33.052 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDEfxdl7NnRNhAAbm5KFLrl3sFGIKx38x+4+A== 2023-09-30T01:54:33.052 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.3] 2023-09-30T01:54:33.052 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDPfxdlhTVCKxAAev9OseT/PoFF0pPpWgS60Q== 2023-09-30T01:54:33.052 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.4] 2023-09-30T01:54:33.052 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDSfxdlaDXPKBAAJmFgO+bfRyzpVoW6EPjvuA== 2023-09-30T01:54:33.052 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.5] 2023-09-30T01:54:33.052 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDVfxdlRL1mJhAAfZmuwQ3tzZovV4gykf5jAw== 2023-09-30T01:54:33.052 INFO:teuthology.orchestra.run.smithi050.stdout:[client.0] 2023-09-30T01:54:33.052 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDEfxdl6UtYORAA9LgkXSKq1pQqY556VUy2fw== 2023-09-30T01:54:33.053 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:54:33.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:54:33.144 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:54:33.147 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:54:33.196 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:54:33.198 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:54:33.246 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T01:54:33.248 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T01:54:33.296 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:54:33.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:54:33.347 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:54:33.349 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:54:33.397 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:54:33.400 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T01:54:33.448 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T01:54:33.450 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T01:54:33.501 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-30T01:54:33.502 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-30T01:54:33.526 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T01:54:33.619 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-30T01:54:33.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-09-30T01:54:33.709 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T01:54:33.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-09-30T01:54:33.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-09-30T01:54:33.853 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T01:54:33.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-09-30T01:54:33.973 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:54:33.975 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T01:54:34.027 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-30T01:54:34.028 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-30T01:54:34.028 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-09-30T01:54:34.030 INFO:tasks.ceph.mon.a:Started 2023-09-30T01:54:34.031 INFO:tasks.ceph.mon.c:Restarting daemon 2023-09-30T01:54:34.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-09-30T01:54:34.032 INFO:tasks.ceph.mon.c:Started 2023-09-30T01:54:34.033 INFO:tasks.ceph.mon.b:Restarting daemon 2023-09-30T01:54:34.033 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-09-30T01:54:34.070 INFO:tasks.ceph.mon.b:Started 2023-09-30T01:54:34.070 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-30T01:54:34.070 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-30T01:54:34.070 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-09-30T01:54:34.072 INFO:tasks.ceph.mgr.x:Started 2023-09-30T01:54:34.073 INFO:tasks.ceph:Setting crush tunables to default 2023-09-30T01:54:34.073 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd crush tunables default 2023-09-30T01:54:34.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.195+0000 7f64025df700 10 monclient: get_monmap_and_config 2023-09-30T01:54:34.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.195+0000 7f64025df700 10 monclient: build_initial_monmap 2023-09-30T01:54:34.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.195+0000 7f64025df700 10 monclient: monmap: 2023-09-30T01:54:34.196 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:34.196 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:34.196 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:34.196674+0000 2023-09-30T01:54:34.196 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:34.196674+0000 2023-09-30T01:54:34.196 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:34.196 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:34.196 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:34.196 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:34.196 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:34.197 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:34.197 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:34.197 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:34.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.196+0000 7f64025df700 1 Processor -- start 2023-09-30T01:54:34.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.196+0000 7f64025df700 1 -- start start 2023-09-30T01:54:34.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.196+0000 7f64025df700 10 monclient: init 2023-09-30T01:54:34.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.196+0000 7f64025df700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:34.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.196+0000 7f64025df700 10 monclient: _add_conns ranks=[0,2,1,3,4] 2023-09-30T01:54:34.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.196+0000 7f64025df700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63fc096a50 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:34.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.196+0000 7f64025df700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc0a2390 0x7f63fc096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.196+0000 7f64025df700 10 monclient(hunting): picked mon.noname-b con 0x7f63fc0a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.196+0000 7f64025df700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63fc097360 addr v1:172.21.15.50:6789/0 2023-09-30T01:54:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.196+0000 7f64025df700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63fc0980e0 con 0x7f63fc096a50 2023-09-30T01:54:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.196+0000 7f64025df700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63fc132eb0 con 0x7f63fc097360 2023-09-30T01:54:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.196+0000 7f64025df700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.196+0000 7f64025df700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc0991a0 con 0x7f63fc0a2390 2023-09-30T01:54:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.196+0000 7f64025df700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.196+0000 7f64025df700 10 monclient(hunting): authenticate will time out at 1163.261855s 2023-09-30T01:54:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.196+0000 7f63fb7fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc0a2390 0x7f63fc096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:34.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.196+0000 7f63fbfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f63fc096a50 0x7f63fc1327a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42702/0 (socket says 172.21.15.7:42702) 2023-09-30T01:54:34.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.196+0000 7f63fbfff700 1 -- 172.21.15.7:0/1152116951 learned_addr learned my addr 172.21.15.7:0/1152116951 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:54:34.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.197+0000 7f63fb7fe700 10 monclient(hunting): get_auth_request con 0x7f63fc0a2390 auth_method 0 2023-09-30T01:54:34.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.197+0000 7f63fb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:34.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.197+0000 7f63fb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:34.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.197+0000 7f63fb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:34.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.197+0000 7f63fb7fe700 1 -- 172.21.15.7:0/1152116951 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc0a2390 msgr2=0x7f63fc096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:54:34.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.197+0000 7f63fb7fe700 1 -- 172.21.15.7:0/1152116951 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc0a2390 msgr2=0x7f63fc096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:54:34.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.197+0000 7f63fb7fe700 1 --2- 172.21.15.7:0/1152116951 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc0a2390 0x7f63fc096e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:54:34.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.197+0000 7f63fb7fe700 1 --2- 172.21.15.7:0/1152116951 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc0a2390 0x7f63fc096e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:54:34.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.199+0000 7f63fbfff700 1 -- 172.21.15.7:0/1152116951 >> v1:172.21.15.7:6789/0 conn(0x7f63fc096a50 legacy=0x7f63fc1327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:34.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.199+0000 7f6400b7c700 1 -- 172.21.15.7:0/1152116951 >> v1:172.21.15.50:6789/0 conn(0x7f63fc097360 legacy=0x7f63fc139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:34.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.398+0000 7f63fb7fe700 1 --2- 172.21.15.7:0/1152116951 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc0a2390 0x7f63fc096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:34.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.398+0000 7f63fb7fe700 10 monclient(hunting): get_auth_request con 0x7f63fc0a2390 auth_method 0 2023-09-30T01:54:34.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.398+0000 7f63fb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:34.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.398+0000 7f63fb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:34.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.398+0000 7f63fb7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:54:34.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.398+0000 7f63fb7fe700 1 -- 172.21.15.7:0/1152116951 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc0a2390 msgr2=0x7f63fc096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T01:54:34.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.398+0000 7f63fb7fe700 1 -- 172.21.15.7:0/1152116951 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc0a2390 msgr2=0x7f63fc096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:54:34.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.398+0000 7f63fb7fe700 1 --2- 172.21.15.7:0/1152116951 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc0a2390 0x7f63fc096e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:54:34.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.398+0000 7f63fb7fe700 1 --2- 172.21.15.7:0/1152116951 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc0a2390 0x7f63fc096e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T01:54:34.475 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.470+0000 7f63faffd700 1 -- 172.21.15.7:0/1152116951 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510274447 0 0) 0x7f63fc0980e0 con 0x7f63fc096a50 2023-09-30T01:54:34.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.470+0000 7f63faffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:34.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.470+0000 7f63faffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:34.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.470+0000 7f63faffd700 10 monclient(hunting): my global_id is 4098 2023-09-30T01:54:34.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.470+0000 7f63faffd700 1 -- 172.21.15.7:0/1152116951 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63e4003570 con 0x7f63fc096a50 2023-09-30T01:54:34.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.470+0000 7f63faffd700 1 -- 172.21.15.7:0/1152116951 <== mon.1 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 907004180 0 0) 0x7f63fc132eb0 con 0x7f63fc097360 2023-09-30T01:54:34.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.470+0000 7f63faffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:34.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.470+0000 7f63faffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:34.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.470+0000 7f63faffd700 10 monclient(hunting): my global_id is 4099 2023-09-30T01:54:34.476 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.470+0000 7f63faffd700 1 -- 172.21.15.7:0/1152116951 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63fc0980e0 con 0x7f63fc097360 2023-09-30T01:54:34.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 1 -- 172.21.15.7:0/1152116951 <== mon.1 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2081438564 0 0) 0x7f63fc0980e0 con 0x7f63fc097360 2023-09-30T01:54:34.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 1 -- 172.21.15.7:0/1152116951 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc0a2390 msgr2=0x7f63fc096e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:54:34.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 1 --2- 172.21.15.7:0/1152116951 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc0a2390 0x7f63fc096e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:34.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 1 -- 172.21.15.7:0/1152116951 >> v1:172.21.15.7:6789/0 conn(0x7f63fc096a50 legacy=0x7f63fc1327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:34.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 10 monclient: _finish_hunting 0 2023-09-30T01:54:34.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 1 monclient: found mon.noname-c-legacy 2023-09-30T01:54:34.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:34.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.50:6789/0 2023-09-30T01:54:34.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 1 -- 172.21.15.7:0/1152116951 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63fc099480 con 0x7f63fc097360 2023-09-30T01:54:34.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 10 monclient: _finish_auth 0 2023-09-30T01:54:34.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:34.477 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 1 -- 172.21.15.7:0/1152116951 <== mon.1 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4135539532 0 0) 0x7f63f4001af0 con 0x7f63fc097360 2023-09-30T01:54:34.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:34.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-09-30T01:54:34.478 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 10 monclient: dump: 2023-09-30T01:54:34.478 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:34.478 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:34.478 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:34.478 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:34.478 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:34.478 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:34.478 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:34.478 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:34.478 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:34.479 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:34.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] but i'm connected to v1:172.21.15.50:6789/0, reconnecting 2023-09-30T01:54:34.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:34.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 1 -- 172.21.15.7:0/1152116951 >> v1:172.21.15.50:6789/0 conn(0x7f63fc097360 legacy=0x7f63fc139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:34.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:54:34.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 1 --2- 172.21.15.7:0/1152116951 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc0a2390 0x7f63e4003c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:34.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 10 monclient(hunting): picked mon.c con 0x7f63fc0a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:34.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 1 --2- 172.21.15.7:0/1152116951 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63e4004220 0x7f63e4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:34.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 10 monclient(hunting): picked mon.b con 0x7f63e4004220 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:34.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 1 --2- 172.21.15.7:0/1152116951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63e4004e70 0x7f63e40052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:34.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 10 monclient(hunting): picked mon.a con 0x7f63e4004e70 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:34.479 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:34.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 1 -- 172.21.15.7:0/1152116951 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc0991a0 con 0x7f63e4004e70 2023-09-30T01:54:34.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:34.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 1 -- 172.21.15.7:0/1152116951 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e400bd90 con 0x7f63e4004220 2023-09-30T01:54:34.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:34.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 1 -- 172.21.15.7:0/1152116951 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e400bf50 con 0x7f63fc0a2390 2023-09-30T01:54:34.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:34.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63faffd700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:54:34.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f6400b7c700 1 --2- 172.21.15.7:0/1152116951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63e4004e70 0x7f63e40052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:34.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63fbfff700 1 --2- 172.21.15.7:0/1152116951 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc0a2390 0x7f63e4003c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:34.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63fb7fe700 1 --2- 172.21.15.7:0/1152116951 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63e4004220 0x7f63e4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:34.480 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63fb7fe700 10 monclient(hunting): get_auth_request con 0x7f63e4004220 auth_method 0 2023-09-30T01:54:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63fb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63fb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63fb7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:54:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f6400b7c700 10 monclient(hunting): get_auth_request con 0x7f63e4004e70 auth_method 0 2023-09-30T01:54:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f6400b7c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f6400b7c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f6400b7c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:54:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63fbfff700 10 monclient(hunting): get_auth_request con 0x7f63fc0a2390 auth_method 0 2023-09-30T01:54:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63fbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63fbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:34.481 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63fbfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:54:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63fb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63fb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63fb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:54:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f6400b7c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f6400b7c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f6400b7c700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:54:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63fbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63fbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63fbfff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:54:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f6400b7c700 10 monclient(hunting): handle_auth_done global_id 4099 payload 293 2023-09-30T01:54:34.482 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f6400b7c700 1 -- 172.21.15.7:0/1152116951 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc0a2390 msgr2=0x7f63e4003c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:34.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f6400b7c700 1 --2- 172.21.15.7:0/1152116951 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc0a2390 0x7f63e4003c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:34.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f6400b7c700 1 -- 172.21.15.7:0/1152116951 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63e4004220 msgr2=0x7f63e4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:34.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f6400b7c700 1 --2- 172.21.15.7:0/1152116951 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63e4004220 0x7f63e4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:34.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f6400b7c700 10 monclient: _finish_hunting 0 2023-09-30T01:54:34.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f6400b7c700 1 monclient: found mon.a 2023-09-30T01:54:34.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f6400b7c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:34.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f6400b7c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:34.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f6400b7c700 1 -- 172.21.15.7:0/1152116951 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63fc0980e0 con 0x7f63e4004e70 2023-09-30T01:54:34.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f6400b7c700 10 monclient: _finish_auth 0 2023-09-30T01:54:34.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f6400b7c700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:34.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f64025df700 5 monclient: authenticate success, global_id 4099 2023-09-30T01:54:34.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f64025df700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:34.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.471+0000 7f63fb7fe700 1 --2- 172.21.15.7:0/1152116951 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63e4004220 0x7f63e4004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:34.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.474+0000 7f6400b7c700 1 -- 172.21.15.7:0/1152116951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63e4004e70 msgr2=0x7f63e40052e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:34.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.474+0000 7f6400b7c700 1 --2- 172.21.15.7:0/1152116951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63e4004e70 0x7f63e40052e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f63f4002230 tx=0x7f63fc097f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a743e676189bfad server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:34.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.474+0000 7f63faffd700 1 -- 172.21.15.7:0/1152116951 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400a100 con 0x7f63e4004e70 2023-09-30T01:54:34.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.474+0000 7f63faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:34.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.474+0000 7f63faffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:34.484 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.474+0000 7f63faffd700 10 monclient: dump: 2023-09-30T01:54:34.484 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:34.484 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:34.484 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:34.484 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:34.484 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:34.485 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:34.485 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:34.485 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:34.485 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:34.485 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:34.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.474+0000 7f63faffd700 1 -- 172.21.15.7:0/1152116951 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005b90 con 0x7f63e4004e70 2023-09-30T01:54:34.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.474+0000 7f64025df700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:34.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.474+0000 7f63faffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:34.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.474+0000 7f64025df700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:34.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.474+0000 7f63faffd700 1 -- 172.21.15.7:0/1152116951 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40070d0 con 0x7f63e4004e70 2023-09-30T01:54:34.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.474+0000 7f63faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:34.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.474+0000 7f64025df700 4 set_mon_vals no callback set 2023-09-30T01:54:34.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.474+0000 7f63faffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:34.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.474+0000 7f63faffd700 10 monclient: dump: 2023-09-30T01:54:34.486 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:34.486 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:34.486 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:34.486 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:34.486 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:34.486 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:34.486 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:34.486 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:34.486 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:34.487 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.474+0000 7f64025df700 10 monclient: shutdown 2023-09-30T01:54:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.474+0000 7f64025df700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.474+0000 7f64025df700 1 -- 172.21.15.7:0/1152116951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63e4004e70 msgr2=0x7f63e40052e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.474+0000 7f64025df700 1 --2- 172.21.15.7:0/1152116951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63e4004e70 0x7f63e40052e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f63f4002230 tx=0x7f63fc097f70 comp rx=0 tx=0).stop 2023-09-30T01:54:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.475+0000 7f6400b7c700 1 -- 172.21.15.7:0/1152116951 reap_dead start 2023-09-30T01:54:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.475+0000 7f64025df700 1 -- 172.21.15.7:0/1152116951 shutdown_connections 2023-09-30T01:54:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.475+0000 7f64025df700 1 -- 172.21.15.7:0/1152116951 >> 172.21.15.7:0/1152116951 conn(0x7f63fc070cf0 msgr2=0x7f63fc0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.475+0000 7f64025df700 1 -- 172.21.15.7:0/1152116951 shutdown_connections 2023-09-30T01:54:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.475+0000 7f64025df700 1 -- 172.21.15.7:0/1152116951 wait complete. 2023-09-30T01:54:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.475+0000 7f64025df700 10 monclient: build_initial_monmap 2023-09-30T01:54:34.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.475+0000 7f64025df700 10 monclient: monmap: 2023-09-30T01:54:34.488 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:34.488 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:34.488 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:34.488 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:34.488 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:34.488 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:34.488 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:34.488 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:34.488 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:34.488 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:34.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.476+0000 7f64025df700 1 Processor -- start 2023-09-30T01:54:34.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.476+0000 7f64025df700 1 -- start start 2023-09-30T01:54:34.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.476+0000 7f64025df700 10 monclient: init 2023-09-30T01:54:34.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f64025df700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:34.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f64025df700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:54:34.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f64025df700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63e4004e70 0x7f63fc12ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:34.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f64025df700 10 monclient(hunting): picked mon.noname-a con 0x7f63e4004e70 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:34.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f64025df700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc12f4b0 0x7f63fc09e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:34.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f64025df700 10 monclient(hunting): picked mon.noname-c con 0x7f63fc12f4b0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:34.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f64025df700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63fc142630 0x7f63fc09ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:34.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f64025df700 10 monclient(hunting): picked mon.noname-b con 0x7f63fc142630 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:34.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f64025df700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:34.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f64025df700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc1334e0 con 0x7f63e4004e70 2023-09-30T01:54:34.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f64025df700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:34.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f64025df700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc0a47d0 con 0x7f63fc142630 2023-09-30T01:54:34.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f64025df700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:34.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f64025df700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63fc11c090 con 0x7f63fc12f4b0 2023-09-30T01:54:34.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f64025df700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:34.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f64025df700 10 monclient(hunting): authenticate will time out at 1163.542502s 2023-09-30T01:54:34.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63e4004e70 0x7f63fc12ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:34.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fbfff700 1 -- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63e4004e70 msgr2=0x7f63fc12ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-09-30T01:54:34.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fbfff700 1 -- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63e4004e70 msgr2=0x7f63fc12ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-30T01:54:34.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fb7fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc12f4b0 0x7f63fc09e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:34.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63e4004e70 0x7f63fc12ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:54:34.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fbfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63e4004e70 0x7f63fc12ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:54:34.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fb7fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc12f4b0 0x7f63fc09e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33438/0 (socket says 172.21.15.7:33438) 2023-09-30T01:54:34.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fb7fe700 1 -- 172.21.15.7:0/2552687404 learned_addr learned my addr 172.21.15.7:0/2552687404 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:34.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fb7fe700 10 monclient(hunting): get_auth_request con 0x7f63fc12f4b0 auth_method 0 2023-09-30T01:54:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f6400b7c700 1 --2- 172.21.15.7:0/2552687404 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63fc142630 0x7f63fc09ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f6400b7c700 10 monclient(hunting): get_auth_request con 0x7f63fc142630 auth_method 0 2023-09-30T01:54:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f6400b7c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f6400b7c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f6400b7c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:34.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f6400b7c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f6400b7c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f6400b7c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fb7fe700 10 monclient(hunting): handle_auth_done global_id 4103 payload 824 2023-09-30T01:54:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fb7fe700 1 -- 172.21.15.7:0/2552687404 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63fc142630 msgr2=0x7f63fc09ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fb7fe700 1 --2- 172.21.15.7:0/2552687404 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63fc142630 0x7f63fc09ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fb7fe700 1 -- 172.21.15.7:0/2552687404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63e4004e70 msgr2=0x7f63fc12ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-30T01:54:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fb7fe700 1 --2- 172.21.15.7:0/2552687404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63e4004e70 0x7f63fc12ef70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fb7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:54:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fb7fe700 1 monclient: found mon.noname-c 2023-09-30T01:54:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.477+0000 7f63fb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.478+0000 7f63fb7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:54:34.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.478+0000 7f63fb7fe700 1 -- 172.21.15.7:0/2552687404 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63fc0a1030 con 0x7f63fc12f4b0 2023-09-30T01:54:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.478+0000 7f6400b7c700 1 --2- 172.21.15.7:0/2552687404 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63fc142630 0x7f63fc09ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f63fb7fe700 1 -- 172.21.15.7:0/2552687404 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc12f4b0 msgr2=0x7f63fc09e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f63fb7fe700 1 --2- 172.21.15.7:0/2552687404 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc12f4b0 0x7f63fc09e8a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f63f0012b50 tx=0x7f63f0008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f63e3fff700 1 -- 172.21.15.7:0/2552687404 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0010b60 con 0x7f63fc12f4b0 2023-09-30T01:54:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f63e3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f63e3fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f63e3fff700 10 monclient: dump: 2023-09-30T01:54:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:34.497 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:34.498 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:34.498 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:34.498 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:34.498 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:34.498 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:34.498 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:34.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f63e3fff700 10 monclient: _finish_auth 0 2023-09-30T01:54:34.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f63e3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:34.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f63e3fff700 1 -- 172.21.15.7:0/2552687404 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0010cc0 con 0x7f63fc12f4b0 2023-09-30T01:54:34.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f64025df700 5 monclient: authenticate success, global_id 4103 2023-09-30T01:54:34.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f64025df700 10 monclient: _renew_subs 2023-09-30T01:54:34.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f64025df700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:34.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f64025df700 1 -- 172.21.15.7:0/2552687404 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63fc11bdd0 con 0x7f63fc12f4b0 2023-09-30T01:54:34.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f63e3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:34.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f63e3fff700 1 -- 172.21.15.7:0/2552687404 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0015450 con 0x7f63fc12f4b0 2023-09-30T01:54:34.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f63e3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:34.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f63e3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:34.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f63e3fff700 10 monclient: dump: 2023-09-30T01:54:34.499 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:34.499 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:34.499 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:34.499 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:34.499 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:34.499 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:34.500 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:34.500 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:34.500 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:34.500 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:34.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f63fa7fc700 4 set_mon_vals no callback set 2023-09-30T01:54:34.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f64025df700 10 monclient: _renew_subs 2023-09-30T01:54:34.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f64025df700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:34.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f64025df700 1 -- 172.21.15.7:0/2552687404 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63fc136e10 con 0x7f63fc12f4b0 2023-09-30T01:54:34.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f63e3fff700 1 -- 172.21.15.7:0/2552687404 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f63f0005a00 con 0x7f63fc12f4b0 2023-09-30T01:54:34.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.480+0000 7f63e3fff700 1 -- 172.21.15.7:0/2552687404 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f63f0015d10 con 0x7f63fc12f4b0 2023-09-30T01:54:34.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.481+0000 7f64025df700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:34.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.481+0000 7f64025df700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:34.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.481+0000 7f64025df700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:34.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.481+0000 7f64025df700 1 -- 172.21.15.7:0/2552687404 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63fc064c10 con 0x7f63fc12f4b0 2023-09-30T01:54:34.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.483+0000 7f63e3fff700 1 -- 172.21.15.7:0/2552687404 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f63f0022020 con 0x7f63fc12f4b0 2023-09-30T01:54:34.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.483+0000 7f63e3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:34.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.483+0000 7f63e3fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:34.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.558+0000 7f64025df700 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2023-09-30T01:54:34.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.558+0000 7f64025df700 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-09-30T01:54:34.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.558+0000 7f64025df700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:34.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.558+0000 7f64025df700 1 -- 172.21.15.7:0/2552687404 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f63fc04bb40 con 0x7f63fc12f4b0 2023-09-30T01:54:34.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.561+0000 7f63e3fff700 1 -- 172.21.15.7:0/2552687404 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f63f0019030 con 0x7f63fc12f4b0 2023-09-30T01:54:34.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.561+0000 7f63e3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-09-30T01:54:34.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.561+0000 7f63e3fff700 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2023-09-30T01:54:34.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.562+0000 7f64025df700 10 monclient: shutdown 2023-09-30T01:54:34.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.562+0000 7f64025df700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.562+0000 7f64025df700 1 -- 172.21.15.7:0/2552687404 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc12f4b0 msgr2=0x7f63fc09e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.562+0000 7f64025df700 1 --2- 172.21.15.7:0/2552687404 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc12f4b0 0x7f63fc09e8a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f63f0012b50 tx=0x7f63f0008040 comp rx=0 tx=0).stop 2023-09-30T01:54:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.562+0000 7f64025df700 1 -- 172.21.15.7:0/2552687404 shutdown_connections 2023-09-30T01:54:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.562+0000 7f64025df700 1 --2- 172.21.15.7:0/2552687404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63e4004e70 0x7f63fc12ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.562+0000 7f64025df700 1 --2- 172.21.15.7:0/2552687404 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63fc142630 0x7f63fc09ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.562+0000 7f64025df700 1 --2- 172.21.15.7:0/2552687404 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f63fc12f4b0 0x7f63fc09e8a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.562+0000 7f64025df700 1 -- 172.21.15.7:0/2552687404 >> 172.21.15.7:0/2552687404 conn(0x7f63fc070cf0 msgr2=0x7f63fc0a5940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.562+0000 7f64025df700 1 -- 172.21.15.7:0/2552687404 shutdown_connections 2023-09-30T01:54:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:34.562+0000 7f64025df700 1 -- 172.21.15.7:0/2552687404 wait complete. 2023-09-30T01:54:34.564 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2023-09-30T01:54:34.574 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-30T01:54:34.574 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-09-30T01:54:34.574 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-30T01:54:34.599 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-09-30T01:54:34.599 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-30T01:54:34.664 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-09-30T01:54:34.664 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-30T01:54:34.729 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-09-30T01:54:34.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-09-30T01:54:34.754 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-09-30T01:54:34.754 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-09-30T01:54:34.819 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2023-09-30T01:54:34.819 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-09-30T01:54:34.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd new e44e9247-cb86-4a16-ac1f-f021cdf5108f 0 2023-09-30T01:54:35.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.041+0000 7f118cebd700 10 monclient: get_monmap_and_config 2023-09-30T01:54:35.041 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.041+0000 7f118cebd700 10 monclient: build_initial_monmap 2023-09-30T01:54:35.042 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.041+0000 7f118cebd700 10 monclient: monmap: 2023-09-30T01:54:35.042 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-09-30T01:54:35.042 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:35.042 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:35.042241+0000 2023-09-30T01:54:35.042 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:35.042241+0000 2023-09-30T01:54:35.042 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:35.042 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:35.042 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:35.042 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:35.042 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:35.043 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:35.043 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:35.043 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:35.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.041+0000 7f118cebd700 1 Processor -- start 2023-09-30T01:54:35.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.041+0000 7f118cebd700 1 -- start start 2023-09-30T01:54:35.043 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.041+0000 7f118cebd700 10 monclient: init 2023-09-30T01:54:35.053 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118cebd700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:35.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118cebd700 10 monclient: _add_conns ranks=[4,2,0,3,1] 2023-09-30T01:54:35.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118cebd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1188096a50 0x7f11881327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:35.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118cebd700 10 monclient(hunting): picked mon.noname-c con 0x7f1188096a50 addr v2:172.21.15.50:3300/0 2023-09-30T01:54:35.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118cebd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11880a2390 0x7f1188096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:35.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118cebd700 10 monclient(hunting): picked mon.noname-b con 0x7f11880a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:35.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118cebd700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1188097360 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:35.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118cebd700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11880980e0 con 0x7f1188097360 2023-09-30T01:54:35.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118cebd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:35.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118cebd700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11880991a0 con 0x7f1188096a50 2023-09-30T01:54:35.054 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118cebd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:35.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118cebd700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1188099480 con 0x7f11880a2390 2023-09-30T01:54:35.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118cebd700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:35.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118cebd700 10 monclient(hunting): authenticate will time out at 1165.270663s 2023-09-30T01:54:35.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118659c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1188096a50 0x7f11881327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:35.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118659c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1188096a50 0x7f11881327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59692/0 (socket says 172.21.15.50:59692) 2023-09-30T01:54:35.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118659c700 1 -- 172.21.15.50:0/385040810 learned_addr learned my addr 172.21.15.50:0/385040810 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-09-30T01:54:35.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118659c700 10 monclient(hunting): get_auth_request con 0x7f1188096a50 auth_method 0 2023-09-30T01:54:35.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:35.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:35.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:35.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:35.055 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:35.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f118659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:35.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.042+0000 7f1185d9b700 1 --2- 172.21.15.50:0/385040810 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11880a2390 0x7f1188096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:35.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.043+0000 7f1185d9b700 10 monclient(hunting): get_auth_request con 0x7f11880a2390 auth_method 0 2023-09-30T01:54:35.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.043+0000 7f1185d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:35.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.043+0000 7f1185d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:35.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.043+0000 7f1185d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:35.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.043+0000 7f118659c700 10 monclient(hunting): handle_auth_done global_id 4111 payload 274 2023-09-30T01:54:35.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.043+0000 7f118659c700 1 -- 172.21.15.50:0/385040810 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11880a2390 msgr2=0x7f1188096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:35.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.043+0000 7f118659c700 1 --2- 172.21.15.50:0/385040810 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11880a2390 0x7f1188096e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.043+0000 7f118659c700 1 -- 172.21.15.50:0/385040810 >> v1:172.21.15.7:6789/0 conn(0x7f1188097360 legacy=0x7f118812ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:35.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.043+0000 7f118659c700 10 monclient: _finish_hunting 0 2023-09-30T01:54:35.056 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.043+0000 7f118659c700 1 monclient: found mon.noname-c 2023-09-30T01:54:35.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.043+0000 7f118659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:35.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.043+0000 7f118659c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3300/0 2023-09-30T01:54:35.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.043+0000 7f118659c700 1 -- 172.21.15.50:0/385040810 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1188097bf0 con 0x7f1188096a50 2023-09-30T01:54:35.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.045+0000 7f118659c700 1 -- 172.21.15.50:0/385040810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1188096a50 msgr2=0x7f11881327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:35.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.045+0000 7f118659c700 1 --2- 172.21.15.50:0/385040810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1188096a50 0x7f11881327a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f117c0099b0 tx=0x7f117c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b755cfbde81f102b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:35.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.045+0000 7f118559a700 1 -- 172.21.15.50:0/385040810 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c00ba30 con 0x7f1188096a50 2023-09-30T01:54:35.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.045+0000 7f118559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:35.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.045+0000 7f118559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:35.057 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.045+0000 7f118559a700 10 monclient: dump: 2023-09-30T01:54:35.057 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:35.058 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:35.058 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.058 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.058 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:35.058 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:35.058 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:35.058 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:35.058 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:35.058 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:35.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.045+0000 7f118559a700 10 monclient: _finish_auth 0 2023-09-30T01:54:35.058 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.045+0000 7f118559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:35.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.045+0000 7f118559a700 1 -- 172.21.15.50:0/385040810 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117c00bb90 con 0x7f1188096a50 2023-09-30T01:54:35.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.045+0000 7f118cebd700 5 monclient: authenticate success, global_id 4111 2023-09-30T01:54:35.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.045+0000 7f118cebd700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:35.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.045+0000 7f118559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:35.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.045+0000 7f118559a700 1 -- 172.21.15.50:0/385040810 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c011420 con 0x7f1188096a50 2023-09-30T01:54:35.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.045+0000 7f118cebd700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:35.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.045+0000 7f118cebd700 4 set_mon_vals no callback set 2023-09-30T01:54:35.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.045+0000 7f118559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:35.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.045+0000 7f118559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:54:35.059 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.045+0000 7f118559a700 10 monclient: dump: 2023-09-30T01:54:35.059 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:35.059 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:35.059 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.060 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.060 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:35.060 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:35.060 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:35.060 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:35.060 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:35.060 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:35.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.045+0000 7f118cebd700 10 monclient: shutdown 2023-09-30T01:54:35.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.045+0000 7f118cebd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:35.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.045+0000 7f118cebd700 1 -- 172.21.15.50:0/385040810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1188096a50 msgr2=0x7f11881327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:35.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.046+0000 7f118cebd700 1 --2- 172.21.15.50:0/385040810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1188096a50 0x7f11881327a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f117c0099b0 tx=0x7f117c009250 comp rx=0 tx=0).stop 2023-09-30T01:54:35.060 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.046+0000 7f118cebd700 1 -- 172.21.15.50:0/385040810 shutdown_connections 2023-09-30T01:54:35.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.046+0000 7f118cebd700 1 --2- 172.21.15.50:0/385040810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1188096a50 0x7f11881327a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.046+0000 7f118cebd700 1 --2- 172.21.15.50:0/385040810 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f11880a2390 0x7f1188096e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.046+0000 7f118cebd700 1 -- 172.21.15.50:0/385040810 >> 172.21.15.50:0/385040810 conn(0x7f1188070cf0 msgr2=0x7f11880710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:35.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.046+0000 7f118cebd700 1 -- 172.21.15.50:0/385040810 shutdown_connections 2023-09-30T01:54:35.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.046+0000 7f118cebd700 1 -- 172.21.15.50:0/385040810 wait complete. 2023-09-30T01:54:35.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.046+0000 7f118cebd700 10 monclient: build_initial_monmap 2023-09-30T01:54:35.061 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.046+0000 7f118cebd700 10 monclient: monmap: 2023-09-30T01:54:35.061 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-09-30T01:54:35.061 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:35.061 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-09-30T01:54:35.061 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-09-30T01:54:35.062 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:35.062 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:35.062 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:35.062 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:35.062 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:35.062 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:35.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118cebd700 1 Processor -- start 2023-09-30T01:54:35.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118cebd700 1 -- start start 2023-09-30T01:54:35.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118cebd700 10 monclient: init 2023-09-30T01:54:35.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118cebd700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:35.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118cebd700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:54:35.062 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118cebd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1188096a50 0x7f118812dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:35.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118cebd700 10 monclient(hunting): picked mon.noname-a con 0x7f1188096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:35.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118cebd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1188097360 0x7f118809dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:35.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118cebd700 10 monclient(hunting): picked mon.noname-c con 0x7f1188097360 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:35.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118cebd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11880a2390 0x7f118809e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:35.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118cebd700 10 monclient(hunting): picked mon.noname-b con 0x7f11880a2390 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:35.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118cebd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:35.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118cebd700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118812f030 con 0x7f1188096a50 2023-09-30T01:54:35.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118cebd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:35.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118cebd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11880a1530 con 0x7f11880a2390 2023-09-30T01:54:35.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118cebd700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:35.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118cebd700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11880a1250 con 0x7f1188097360 2023-09-30T01:54:35.063 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118cebd700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:35.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118cebd700 10 monclient(hunting): authenticate will time out at 1165.275587s 2023-09-30T01:54:35.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f1186d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11880a2390 0x7f118809e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:35.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f1186d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11880a2390 0x7f118809e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59696/0 (socket says 172.21.15.50:59696) 2023-09-30T01:54:35.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f1186d9d700 1 -- 172.21.15.50:0/3693268284 learned_addr learned my addr 172.21.15.50:0/3693268284 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-09-30T01:54:35.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f1186d9d700 10 monclient(hunting): get_auth_request con 0x7f11880a2390 auth_method 0 2023-09-30T01:54:35.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f1186d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:35.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f1186d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:35.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f1186d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:35.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f1186d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:35.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f1186d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:35.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f1186d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:35.064 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f1185d9b700 1 --2- 172.21.15.50:0/3693268284 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1188097360 0x7f118809dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:35.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f1185d9b700 10 monclient(hunting): get_auth_request con 0x7f1188097360 auth_method 0 2023-09-30T01:54:35.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f1185d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:35.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f1185d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:35.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f1185d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:35.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118659c700 1 --2- 172.21.15.50:0/3693268284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1188096a50 0x7f118812dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:35.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118659c700 10 monclient(hunting): get_auth_request con 0x7f1188096a50 auth_method 0 2023-09-30T01:54:35.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:35.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:35.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f118659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:35.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f1186d9d700 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2023-09-30T01:54:35.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f1186d9d700 1 -- 172.21.15.50:0/3693268284 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1188097360 msgr2=0x7f118809dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:35.065 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f1186d9d700 1 --2- 172.21.15.50:0/3693268284 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1188097360 0x7f118809dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.047+0000 7f1186d9d700 1 -- 172.21.15.50:0/3693268284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1188096a50 msgr2=0x7f118812dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:35.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.048+0000 7f1186d9d700 1 --2- 172.21.15.50:0/3693268284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1188096a50 0x7f118812dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.048+0000 7f1186d9d700 10 monclient: _finish_hunting 0 2023-09-30T01:54:35.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.048+0000 7f1186d9d700 1 monclient: found mon.noname-b 2023-09-30T01:54:35.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.048+0000 7f1186d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:35.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.048+0000 7f1186d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3300/0 2023-09-30T01:54:35.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.048+0000 7f1186d9d700 1 -- 172.21.15.50:0/3693268284 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c0094f0 con 0x7f11880a2390 2023-09-30T01:54:35.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.048+0000 7f1185d9b700 1 --2- 172.21.15.50:0/3693268284 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1188097360 0x7f118809dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:54:35.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f1186d9d700 1 -- 172.21.15.50:0/3693268284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11880a2390 msgr2=0x7f118809e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:35.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f1186d9d700 1 --2- 172.21.15.50:0/3693268284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11880a2390 0x7f118809e160 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f118000c930 tx=0x7f11800026e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:35.066 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f11767fc700 1 -- 172.21.15.50:0/3693268284 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118000ce70 con 0x7f11880a2390 2023-09-30T01:54:35.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f11767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:35.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f11767fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:54:35.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f11767fc700 10 monclient: dump: 2023-09-30T01:54:35.067 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:35.067 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:35.067 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.067 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.067 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:35.067 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:35.067 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:35.067 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:35.067 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:35.067 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:35.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f11767fc700 10 monclient: _finish_auth 0 2023-09-30T01:54:35.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f11767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:35.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f118cebd700 5 monclient: authenticate success, global_id 4114 2023-09-30T01:54:35.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f118cebd700 10 monclient: _renew_subs 2023-09-30T01:54:35.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f11767fc700 1 -- 172.21.15.50:0/3693268284 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118000a9f0 con 0x7f11880a2390 2023-09-30T01:54:35.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f118cebd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:35.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f118cebd700 1 -- 172.21.15.50:0/3693268284 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f118809f7b0 con 0x7f11880a2390 2023-09-30T01:54:35.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f11767fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:35.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f11767fc700 1 -- 172.21.15.50:0/3693268284 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118000add0 con 0x7f11880a2390 2023-09-30T01:54:35.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f11767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:35.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f11767fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:54:35.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f11767fc700 10 monclient: dump: 2023-09-30T01:54:35.069 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:35.069 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:35.069 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.069 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.069 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:35.069 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:35.069 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:35.069 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:35.069 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:35.069 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:35.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f1184d99700 4 set_mon_vals no callback set 2023-09-30T01:54:35.069 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f118cebd700 10 monclient: _renew_subs 2023-09-30T01:54:35.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f118cebd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:35.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f118cebd700 1 -- 172.21.15.50:0/3693268284 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1188136ec0 con 0x7f11880a2390 2023-09-30T01:54:35.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f11767fc700 1 -- 172.21.15.50:0/3693268284 <== mon.1 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f11800128b0 con 0x7f11880a2390 2023-09-30T01:54:35.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.050+0000 7f11767fc700 1 -- 172.21.15.50:0/3693268284 <== mon.1 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f118001b980 con 0x7f11880a2390 2023-09-30T01:54:35.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.051+0000 7f118cebd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:35.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.051+0000 7f118cebd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:35.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.051+0000 7f118cebd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:35.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.051+0000 7f118cebd700 1 -- 172.21.15.50:0/3693268284 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11880a48f0 con 0x7f11880a2390 2023-09-30T01:54:35.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.053+0000 7f11767fc700 1 -- 172.21.15.50:0/3693268284 <== mon.1 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f1180016020 con 0x7f11880a2390 2023-09-30T01:54:35.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.053+0000 7f11767fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:35.070 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.053+0000 7f11767fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:35.136 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.136+0000 7f118cebd700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e44e9247-cb86-4a16-ac1f-f021cdf5108f", "id": 0}] 2023-09-30T01:54:35.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.136+0000 7f118cebd700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e44e9247-cb86-4a16-ac1f-f021cdf5108f", "id": 0}] 2023-09-30T01:54:35.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.137+0000 7f118cebd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:35.137 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.137+0000 7f118cebd700 1 -- 172.21.15.50:0/3693268284 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e44e9247-cb86-4a16-ac1f-f021cdf5108f", "id": 0} v 0) v1 -- 0x7f1188064c10 con 0x7f11880a2390 2023-09-30T01:54:35.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.147+0000 7f11767fc700 1 -- 172.21.15.50:0/3693268284 <== mon.1 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e44e9247-cb86-4a16-ac1f-f021cdf5108f", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1180012d40 con 0x7f11880a2390 2023-09-30T01:54:35.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.147+0000 7f11767fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e44e9247-cb86-4a16-ac1f-f021cdf5108f", "id": 0}] 2023-09-30T01:54:35.147 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.147+0000 7f11767fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:35.147 INFO:teuthology.orchestra.run.smithi050.stdout:0 2023-09-30T01:54:35.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.148+0000 7f118cebd700 10 monclient: shutdown 2023-09-30T01:54:35.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.148+0000 7f118cebd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:35.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.148+0000 7f118cebd700 1 -- 172.21.15.50:0/3693268284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11880a2390 msgr2=0x7f118809e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:35.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.148+0000 7f118cebd700 1 --2- 172.21.15.50:0/3693268284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11880a2390 0x7f118809e160 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f118000c930 tx=0x7f11800026e0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.148 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.148+0000 7f118cebd700 1 -- 172.21.15.50:0/3693268284 shutdown_connections 2023-09-30T01:54:35.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.148+0000 7f118cebd700 1 --2- 172.21.15.50:0/3693268284 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1188096a50 0x7f118812dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.148+0000 7f118cebd700 1 --2- 172.21.15.50:0/3693268284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11880a2390 0x7f118809e160 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.148+0000 7f118cebd700 1 --2- 172.21.15.50:0/3693268284 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1188097360 0x7f118809dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.148+0000 7f118cebd700 1 -- 172.21.15.50:0/3693268284 >> 172.21.15.50:0/3693268284 conn(0x7f1188070cf0 msgr2=0x7f11881310a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:35.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.148+0000 7f118cebd700 1 -- 172.21.15.50:0/3693268284 shutdown_connections 2023-09-30T01:54:35.149 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.148+0000 7f118cebd700 1 -- 172.21.15.50:0/3693268284 wait complete. 2023-09-30T01:54:35.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd new e30730fd-1e49-44bf-bdc9-087e23727df0 1 2023-09-30T01:54:35.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.277+0000 7ff6ef0c3700 10 monclient: get_monmap_and_config 2023-09-30T01:54:35.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.277+0000 7ff6ef0c3700 10 monclient: build_initial_monmap 2023-09-30T01:54:35.278 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.277+0000 7ff6ef0c3700 10 monclient: monmap: 2023-09-30T01:54:35.278 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-09-30T01:54:35.278 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:35.278 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:35.278712+0000 2023-09-30T01:54:35.278 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:35.278712+0000 2023-09-30T01:54:35.279 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:35.279 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:35.279 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:35.279 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:35.279 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:35.279 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:35.279 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:35.279 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:35.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.278+0000 7ff6ef0c3700 1 Processor -- start 2023-09-30T01:54:35.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.278+0000 7ff6ef0c3700 1 -- start start 2023-09-30T01:54:35.279 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.278+0000 7ff6ef0c3700 10 monclient: init 2023-09-30T01:54:35.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ef0c3700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:35.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ef0c3700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2023-09-30T01:54:35.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ef0c3700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff6e812ff60 0x7ff6e8098500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:35.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ef0c3700 10 monclient(hunting): picked mon.noname-b con 0x7ff6e812ff60 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:35.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ef0c3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff6e8130ab0 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:35.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ef0c3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff6e81320d0 addr v1:172.21.15.50:6789/0 2023-09-30T01:54:35.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ef0c3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e8096ac0 con 0x7ff6e8130ab0 2023-09-30T01:54:35.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ef0c3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e8099150 con 0x7ff6e81320d0 2023-09-30T01:54:35.288 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ef0c3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:35.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ef0c3700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e8130330 con 0x7ff6e812ff60 2023-09-30T01:54:35.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ef0c3700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:35.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ef0c3700 10 monclient(hunting): authenticate will time out at 1165.507123s 2023-09-30T01:54:35.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ee8c2700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff6e81320d0 0x7ff6e8132510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:45368/0 (socket says 172.21.15.50:45368) 2023-09-30T01:54:35.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ee8c2700 1 -- 172.21.15.50:0/3201075380 learned_addr learned my addr 172.21.15.50:0/3201075380 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-09-30T01:54:35.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ee8c2700 1 -- 172.21.15.50:0/3201075380 >> v1:172.21.15.50:6789/0 conn(0x7ff6e81320d0 legacy=0x7ff6e8132510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send send error: (32) Broken pipe 2023-09-30T01:54:35.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ee0c1700 1 --2- 172.21.15.50:0/3201075380 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff6e812ff60 0x7ff6e8098500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:35.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ee0c1700 10 monclient(hunting): get_auth_request con 0x7ff6e812ff60 auth_method 0 2023-09-30T01:54:35.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ee0c1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:35.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ee0c1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:35.289 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ee0c1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:35.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ee0c1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:35.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ee0c1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:35.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ee0c1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:35.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ee0c1700 10 monclient(hunting): handle_auth_done global_id 4112 payload 274 2023-09-30T01:54:35.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ee0c1700 1 -- 172.21.15.50:0/3201075380 >> v1:172.21.15.50:6789/0 conn(0x7ff6e81320d0 legacy=0x7ff6e8132510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:54:35.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ee0c1700 1 -- 172.21.15.50:0/3201075380 >> v1:172.21.15.7:6789/0 conn(0x7ff6e8130ab0 legacy=0x7ff6e8098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:35.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ee0c1700 10 monclient: _finish_hunting 0 2023-09-30T01:54:35.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ee0c1700 1 monclient: found mon.noname-b 2023-09-30T01:54:35.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ee0c1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:35.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ee0c1700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:54:35.290 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.279+0000 7ff6ee0c1700 1 -- 172.21.15.50:0/3201075380 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e8096cf0 con 0x7ff6e812ff60 2023-09-30T01:54:35.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ee0c1700 1 -- 172.21.15.50:0/3201075380 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff6e812ff60 msgr2=0x7ff6e8098500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:35.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ee0c1700 1 --2- 172.21.15.50:0/3201075380 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff6e812ff60 0x7ff6e8098500 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8009910 tx=0x7ff6d80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84c57bf76f8a605 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:35.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ed0bf700 1 -- 172.21.15.50:0/3201075380 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d800b9f0 con 0x7ff6e812ff60 2023-09-30T01:54:35.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ed0bf700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:35.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ed0bf700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:54:35.291 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ed0bf700 10 monclient: dump: 2023-09-30T01:54:35.291 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:35.291 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:35.291 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.291 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.291 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:35.292 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:35.292 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:35.292 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:35.292 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:35.292 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:35.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ed0bf700 10 monclient: _finish_auth 0 2023-09-30T01:54:35.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ed0bf700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:35.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ed0bf700 1 -- 172.21.15.50:0/3201075380 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d800bb50 con 0x7ff6e812ff60 2023-09-30T01:54:35.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ef0c3700 5 monclient: authenticate success, global_id 4112 2023-09-30T01:54:35.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ef0c3700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:35.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ed0bf700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:35.292 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ef0c3700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:35.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ed0bf700 1 -- 172.21.15.50:0/3201075380 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8011420 con 0x7ff6e812ff60 2023-09-30T01:54:35.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ed0bf700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:35.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ef0c3700 4 set_mon_vals no callback set 2023-09-30T01:54:35.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ed0bf700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:35.293 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ed0bf700 10 monclient: dump: 2023-09-30T01:54:35.293 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:35.293 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:35.293 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.293 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.293 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:35.293 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:35.293 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:35.294 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:35.294 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:35.294 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:35.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ef0c3700 10 monclient: shutdown 2023-09-30T01:54:35.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ef0c3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:35.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ef0c3700 1 -- 172.21.15.50:0/3201075380 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff6e812ff60 msgr2=0x7ff6e8098500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:35.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ef0c3700 1 --2- 172.21.15.50:0/3201075380 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff6e812ff60 0x7ff6e8098500 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8009910 tx=0x7ff6d80091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ef0c3700 1 -- 172.21.15.50:0/3201075380 shutdown_connections 2023-09-30T01:54:35.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ef0c3700 1 --2- 172.21.15.50:0/3201075380 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff6e812ff60 0x7ff6e8098500 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ef0c3700 1 -- 172.21.15.50:0/3201075380 >> 172.21.15.50:0/3201075380 conn(0x7ff6e8071870 msgr2=0x7ff6e80a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:35.294 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ef0c3700 1 -- 172.21.15.50:0/3201075380 shutdown_connections 2023-09-30T01:54:35.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.282+0000 7ff6ef0c3700 1 -- 172.21.15.50:0/3201075380 wait complete. 2023-09-30T01:54:35.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.283+0000 7ff6ef0c3700 10 monclient: build_initial_monmap 2023-09-30T01:54:35.295 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.283+0000 7ff6ef0c3700 10 monclient: monmap: 2023-09-30T01:54:35.295 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-09-30T01:54:35.295 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:35.295 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-09-30T01:54:35.295 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-09-30T01:54:35.295 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:35.295 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:35.295 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:35.295 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:35.296 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:35.296 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:35.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.283+0000 7ff6ef0c3700 1 Processor -- start 2023-09-30T01:54:35.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.283+0000 7ff6ef0c3700 1 -- start start 2023-09-30T01:54:35.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.283+0000 7ff6ef0c3700 10 monclient: init 2023-09-30T01:54:35.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ef0c3700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:35.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ef0c3700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:54:35.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ef0c3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e812ff60 0x7ff6e811c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:35.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ef0c3700 10 monclient(hunting): picked mon.noname-b con 0x7ff6e812ff60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:35.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ef0c3700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff6e8130ab0 0x7ff6e811c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:35.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ef0c3700 10 monclient(hunting): picked mon.noname-c con 0x7ff6e8130ab0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:35.296 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ef0c3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6e81320d0 0x7ff6e811cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:35.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ef0c3700 10 monclient(hunting): picked mon.noname-a con 0x7ff6e81320d0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:35.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ef0c3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:35.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ef0c3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e812edd0 con 0x7ff6e81320d0 2023-09-30T01:54:35.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ef0c3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:35.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ef0c3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e806b050 con 0x7ff6e812ff60 2023-09-30T01:54:35.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ef0c3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:35.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ef0c3700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e811fa80 con 0x7ff6e8130ab0 2023-09-30T01:54:35.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ef0c3700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:35.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ef0c3700 10 monclient(hunting): authenticate will time out at 1165.512321s 2023-09-30T01:54:35.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee0c1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e812ff60 0x7ff6e811c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:35.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee0c1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e812ff60 0x7ff6e811c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59700/0 (socket says 172.21.15.50:59700) 2023-09-30T01:54:35.297 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee0c1700 1 -- 172.21.15.50:0/1735649026 learned_addr learned my addr 172.21.15.50:0/1735649026 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-09-30T01:54:35.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee0c1700 10 monclient(hunting): get_auth_request con 0x7ff6e812ff60 auth_method 0 2023-09-30T01:54:35.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee0c1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:35.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee0c1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:35.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee0c1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:35.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee8c2700 1 --2- 172.21.15.50:0/1735649026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6e81320d0 0x7ff6e811cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:35.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ed8c0700 1 --2- 172.21.15.50:0/1735649026 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff6e8130ab0 0x7ff6e811c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:35.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee8c2700 10 monclient(hunting): get_auth_request con 0x7ff6e81320d0 auth_method 0 2023-09-30T01:54:35.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee8c2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:35.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee8c2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:35.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee8c2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:35.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee0c1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:35.298 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee0c1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:35.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee0c1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:35.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ed8c0700 10 monclient(hunting): get_auth_request con 0x7ff6e8130ab0 auth_method 0 2023-09-30T01:54:35.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ed8c0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:35.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ed8c0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:35.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ed8c0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:35.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee8c2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:35.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee8c2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:35.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee8c2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:35.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ed8c0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:35.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ed8c0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:35.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ed8c0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:35.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee0c1700 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2023-09-30T01:54:35.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee0c1700 1 -- 172.21.15.50:0/1735649026 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff6e8130ab0 msgr2=0x7ff6e811c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:35.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee0c1700 1 --2- 172.21.15.50:0/1735649026 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff6e8130ab0 0x7ff6e811c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee0c1700 1 -- 172.21.15.50:0/1735649026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6e81320d0 msgr2=0x7ff6e811cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:35.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee0c1700 1 --2- 172.21.15.50:0/1735649026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6e81320d0 0x7ff6e811cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee0c1700 10 monclient: _finish_hunting 0 2023-09-30T01:54:35.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee0c1700 1 monclient: found mon.noname-b 2023-09-30T01:54:35.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee0c1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:35.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee0c1700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3300/0 2023-09-30T01:54:35.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.284+0000 7ff6ee0c1700 1 -- 172.21.15.50:0/1735649026 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d8009420 con 0x7ff6e812ff60 2023-09-30T01:54:35.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.286+0000 7ff6ee0c1700 1 -- 172.21.15.50:0/1735649026 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e812ff60 msgr2=0x7ff6e811c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:35.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.286+0000 7ff6ee0c1700 1 --2- 172.21.15.50:0/1735649026 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e812ff60 0x7ff6e811c320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8009fd0 tx=0x7ff6d8016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:35.300 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6de7fc700 1 -- 172.21.15.50:0/1735649026 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d800bed0 con 0x7ff6e812ff60 2023-09-30T01:54:35.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6de7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:35.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6de7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:54:35.301 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6de7fc700 10 monclient: dump: 2023-09-30T01:54:35.301 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:35.301 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:35.301 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.301 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.301 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:35.301 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:35.301 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:35.301 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:35.301 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:35.302 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:35.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6de7fc700 10 monclient: _finish_auth 0 2023-09-30T01:54:35.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6de7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:35.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6de7fc700 1 -- 172.21.15.50:0/1735649026 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6d8020070 con 0x7ff6e812ff60 2023-09-30T01:54:35.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6ef0c3700 5 monclient: authenticate success, global_id 4117 2023-09-30T01:54:35.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6ef0c3700 10 monclient: _renew_subs 2023-09-30T01:54:35.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6ef0c3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:35.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6ef0c3700 1 -- 172.21.15.50:0/1735649026 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e811f7a0 con 0x7ff6e812ff60 2023-09-30T01:54:35.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6de7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:35.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6de7fc700 1 -- 172.21.15.50:0/1735649026 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d801b6a0 con 0x7ff6e812ff60 2023-09-30T01:54:35.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6de7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:35.302 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6de7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:54:35.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6de7fc700 10 monclient: dump: 2023-09-30T01:54:35.303 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:35.303 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:35.303 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.303 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.303 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:35.303 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:35.303 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:35.303 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:35.303 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:35.303 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:35.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6ec8be700 4 set_mon_vals no callback set 2023-09-30T01:54:35.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6ef0c3700 10 monclient: _renew_subs 2023-09-30T01:54:35.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6ef0c3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:35.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6ef0c3700 1 -- 172.21.15.50:0/1735649026 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e812f830 con 0x7ff6e812ff60 2023-09-30T01:54:35.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6de7fc700 1 -- 172.21.15.50:0/1735649026 <== mon.1 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7ff6d8022030 con 0x7ff6e812ff60 2023-09-30T01:54:35.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.287+0000 7ff6de7fc700 1 -- 172.21.15.50:0/1735649026 <== mon.1 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7ff6d801c9e0 con 0x7ff6e812ff60 2023-09-30T01:54:35.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.288+0000 7ff6ef0c3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:35.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.288+0000 7ff6ef0c3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:35.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.288+0000 7ff6ef0c3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:35.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.288+0000 7ff6ef0c3700 1 -- 172.21.15.50:0/1735649026 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e812f830 con 0x7ff6e812ff60 2023-09-30T01:54:35.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.290+0000 7ff6de7fc700 1 -- 172.21.15.50:0/1735649026 <== mon.1 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7ff6d801cbf0 con 0x7ff6e812ff60 2023-09-30T01:54:35.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.290+0000 7ff6de7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:35.306 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.290+0000 7ff6de7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:35.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.391+0000 7ff6ef0c3700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e30730fd-1e49-44bf-bdc9-087e23727df0", "id": 1}] 2023-09-30T01:54:35.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.391+0000 7ff6ef0c3700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e30730fd-1e49-44bf-bdc9-087e23727df0", "id": 1}] 2023-09-30T01:54:35.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.391+0000 7ff6ef0c3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:35.391 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.391+0000 7ff6ef0c3700 1 -- 172.21.15.50:0/1735649026 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e30730fd-1e49-44bf-bdc9-087e23727df0", "id": 1} v 0) v1 -- 0x7ff6e812f830 con 0x7ff6e812ff60 2023-09-30T01:54:35.700 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:35.700+0000 7fc92c365180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T01:54:35.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.707+0000 7ff6de7fc700 1 -- 172.21.15.50:0/1735649026 <== mon.1 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e30730fd-1e49-44bf-bdc9-087e23727df0", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff6d801cd90 con 0x7ff6e812ff60 2023-09-30T01:54:35.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.707+0000 7ff6de7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e30730fd-1e49-44bf-bdc9-087e23727df0", "id": 1}] 2023-09-30T01:54:35.707 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.707+0000 7ff6de7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:35.707 INFO:teuthology.orchestra.run.smithi050.stdout:1 2023-09-30T01:54:35.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.708+0000 7ff6ef0c3700 10 monclient: shutdown 2023-09-30T01:54:35.708 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.708+0000 7ff6ef0c3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:35.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.708+0000 7ff6ef0c3700 1 -- 172.21.15.50:0/1735649026 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e812ff60 msgr2=0x7ff6e811c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:35.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.708+0000 7ff6ef0c3700 1 --2- 172.21.15.50:0/1735649026 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e812ff60 0x7ff6e811c320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8009fd0 tx=0x7ff6d8016000 comp rx=0 tx=0).stop 2023-09-30T01:54:35.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.708+0000 7ff6ef0c3700 1 -- 172.21.15.50:0/1735649026 shutdown_connections 2023-09-30T01:54:35.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.708+0000 7ff6ef0c3700 1 --2- 172.21.15.50:0/1735649026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff6e81320d0 0x7ff6e811cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.708+0000 7ff6ef0c3700 1 --2- 172.21.15.50:0/1735649026 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff6e812ff60 0x7ff6e811c320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.708+0000 7ff6ef0c3700 1 --2- 172.21.15.50:0/1735649026 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff6e8130ab0 0x7ff6e811c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.708+0000 7ff6ef0c3700 1 -- 172.21.15.50:0/1735649026 >> 172.21.15.50:0/1735649026 conn(0x7ff6e8071870 msgr2=0x7ff6e80a3570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:35.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.708+0000 7ff6ef0c3700 1 -- 172.21.15.50:0/1735649026 shutdown_connections 2023-09-30T01:54:35.709 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.708+0000 7ff6ef0c3700 1 -- 172.21.15.50:0/1735649026 wait complete. 2023-09-30T01:54:35.720 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd new b39c552e-f5cc-4e4f-90a3-5aa5f5127a27 2 2023-09-30T01:54:35.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.837+0000 7f85a115e700 10 monclient: get_monmap_and_config 2023-09-30T01:54:35.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.837+0000 7f85a115e700 10 monclient: build_initial_monmap 2023-09-30T01:54:35.837 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.837+0000 7f85a115e700 10 monclient: monmap: 2023-09-30T01:54:35.837 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-09-30T01:54:35.838 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:35.838 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:35.838162+0000 2023-09-30T01:54:35.838 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:35.838162+0000 2023-09-30T01:54:35.838 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:35.838 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:35.838 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:35.838 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:35.838 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:35.838 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:35.838 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:35.838 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:35.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.837+0000 7f85a115e700 1 Processor -- start 2023-09-30T01:54:35.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.837+0000 7f85a115e700 1 -- start start 2023-09-30T01:54:35.839 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.837+0000 7f85a115e700 10 monclient: init 2023-09-30T01:54:35.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f85a115e700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:35.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f85a115e700 10 monclient: _add_conns ranks=[0,3,1,4,2] 2023-09-30T01:54:35.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f85a115e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f859c096a50 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:35.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f85a115e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f859c0a2390 0x7f859c096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:35.844 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f85a115e700 10 monclient(hunting): picked mon.noname-a con 0x7f859c0a2390 addr v2:172.21.15.7:3300/0 2023-09-30T01:54:35.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f85a115e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f859c097360 addr v1:172.21.15.50:6789/0 2023-09-30T01:54:35.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f85a115e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f859c0980e0 con 0x7f859c096a50 2023-09-30T01:54:35.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f85a115e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f859c132eb0 con 0x7f859c097360 2023-09-30T01:54:35.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f85a115e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:35.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f85a115e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f859c0991a0 con 0x7f859c0a2390 2023-09-30T01:54:35.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f85a115e700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:35.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f85a115e700 10 monclient(hunting): authenticate will time out at 1166.066398s 2023-09-30T01:54:35.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f859b59e700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f859c097360 0x7f859c139720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:45376/0 (socket says 172.21.15.50:45376) 2023-09-30T01:54:35.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f859b59e700 1 -- 172.21.15.50:0/3573175562 learned_addr learned my addr 172.21.15.50:0/3573175562 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-09-30T01:54:35.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f859a59c700 1 --2- 172.21.15.50:0/3573175562 >> v2:172.21.15.7:3300/0 conn(0x7f859c0a2390 0x7f859c096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:35.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f859a59c700 10 monclient(hunting): get_auth_request con 0x7f859c0a2390 auth_method 0 2023-09-30T01:54:35.845 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f859a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:35.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f859a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:35.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f859a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:35.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f859a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:35.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f859a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:35.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f859a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:35.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f859a59c700 10 monclient(hunting): handle_auth_done global_id 4113 payload 274 2023-09-30T01:54:35.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f859a59c700 1 -- 172.21.15.50:0/3573175562 >> v1:172.21.15.50:6789/0 conn(0x7f859c097360 legacy=0x7f859c139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:35.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f859a59c700 1 -- 172.21.15.50:0/3573175562 >> v1:172.21.15.7:6789/0 conn(0x7f859c096a50 legacy=0x7f859c1327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:35.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f859a59c700 10 monclient: _finish_hunting 0 2023-09-30T01:54:35.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f859a59c700 1 monclient: found mon.noname-a 2023-09-30T01:54:35.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f859a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:35.846 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f859a59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:35.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.838+0000 7f859a59c700 1 -- 172.21.15.50:0/3573175562 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f859c099480 con 0x7f859c0a2390 2023-09-30T01:54:35.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f859a59c700 1 -- 172.21.15.50:0/3573175562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f859c0a2390 msgr2=0x7f859c096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:35.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f859a59c700 1 --2- 172.21.15.50:0/3573175562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f859c0a2390 0x7f859c096e20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f85900098e0 tx=0x7f85900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5874623f1fec4630 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:35.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f8599d9b700 1 -- 172.21.15.50:0/3573175562 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859000b9f0 con 0x7f859c0a2390 2023-09-30T01:54:35.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f8599d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:35.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f8599d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:35.847 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f8599d9b700 10 monclient: dump: 2023-09-30T01:54:35.847 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:35.847 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:35.847 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.847 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.847 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:35.848 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:35.848 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:35.848 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:35.848 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:35.848 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:35.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f8599d9b700 10 monclient: _finish_auth 0 2023-09-30T01:54:35.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f8599d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:35.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f8599d9b700 1 -- 172.21.15.50:0/3573175562 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f859000bb50 con 0x7f859c0a2390 2023-09-30T01:54:35.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f85a115e700 5 monclient: authenticate success, global_id 4113 2023-09-30T01:54:35.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f85a115e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:35.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f8599d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:35.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f8599d9b700 1 -- 172.21.15.50:0/3573175562 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8590011420 con 0x7f859c0a2390 2023-09-30T01:54:35.848 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f85a115e700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:35.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f85a115e700 4 set_mon_vals no callback set 2023-09-30T01:54:35.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f8599d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:35.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f8599d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:35.849 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f8599d9b700 10 monclient: dump: 2023-09-30T01:54:35.849 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:35.849 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:35.849 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.849 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.849 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:35.849 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:35.849 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:35.849 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:35.850 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:35.850 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:35.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f85a115e700 10 monclient: shutdown 2023-09-30T01:54:35.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f85a115e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:35.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f85a115e700 1 -- 172.21.15.50:0/3573175562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f859c0a2390 msgr2=0x7f859c096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:35.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.841+0000 7f85a115e700 1 --2- 172.21.15.50:0/3573175562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f859c0a2390 0x7f859c096e20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f85900098e0 tx=0x7f85900091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.842+0000 7f85a115e700 1 -- 172.21.15.50:0/3573175562 shutdown_connections 2023-09-30T01:54:35.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.842+0000 7f85a115e700 1 --2- 172.21.15.50:0/3573175562 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f859c0a2390 0x7f859c096e20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.842+0000 7f85a115e700 1 -- 172.21.15.50:0/3573175562 >> 172.21.15.50:0/3573175562 conn(0x7f859c070cf0 msgr2=0x7f859c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:35.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.842+0000 7f85a115e700 1 -- 172.21.15.50:0/3573175562 shutdown_connections 2023-09-30T01:54:35.850 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.842+0000 7f85a115e700 1 -- 172.21.15.50:0/3573175562 wait complete. 2023-09-30T01:54:35.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.842+0000 7f85a115e700 10 monclient: build_initial_monmap 2023-09-30T01:54:35.851 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.842+0000 7f85a115e700 10 monclient: monmap: 2023-09-30T01:54:35.851 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-09-30T01:54:35.851 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:35.851 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-09-30T01:54:35.851 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-09-30T01:54:35.851 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:35.851 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:35.851 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:35.851 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:35.851 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:35.851 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:35.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f85a115e700 1 Processor -- start 2023-09-30T01:54:35.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f85a115e700 1 -- start start 2023-09-30T01:54:35.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f85a115e700 10 monclient: init 2023-09-30T01:54:35.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f85a115e700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:35.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f85a115e700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:54:35.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f85a115e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f859c096a50 0x7f859c142320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:35.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f85a115e700 10 monclient(hunting): picked mon.noname-a con 0x7f859c096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:35.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f85a115e700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f859c097360 0x7f859c142860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:35.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f85a115e700 10 monclient(hunting): picked mon.noname-c con 0x7f859c097360 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:35.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f85a115e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f859c0a2390 0x7f859c09bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:35.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f85a115e700 10 monclient(hunting): picked mon.noname-b con 0x7f859c0a2390 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:35.852 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f85a115e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:35.853 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:35.843+0000 7fc92c365180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T01:54:35.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f85a115e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859c09f440 con 0x7f859c096a50 2023-09-30T01:54:35.857 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f85a115e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:35.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f85a115e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859c09e300 con 0x7f859c0a2390 2023-09-30T01:54:35.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f85a115e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:35.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f85a115e700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f859c1322d0 con 0x7f859c097360 2023-09-30T01:54:35.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f85a115e700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:35.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f85a115e700 10 monclient(hunting): authenticate will time out at 1166.071624s 2023-09-30T01:54:35.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f859b59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f859c0a2390 0x7f859c09bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:35.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f859b59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f859c0a2390 0x7f859c09bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59716/0 (socket says 172.21.15.50:59716) 2023-09-30T01:54:35.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f859b59e700 1 -- 172.21.15.50:0/1491950891 learned_addr learned my addr 172.21.15.50:0/1491950891 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-09-30T01:54:35.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f859b59e700 10 monclient(hunting): get_auth_request con 0x7f859c0a2390 auth_method 0 2023-09-30T01:54:35.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f859b59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:35.858 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f859b59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:35.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f859b59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:35.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f859b59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:35.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f859b59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:35.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f859b59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:35.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f859a59c700 1 --2- 172.21.15.50:0/1491950891 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f859c097360 0x7f859c142860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:35.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f859ad9d700 1 --2- 172.21.15.50:0/1491950891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f859c096a50 0x7f859c142320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:35.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f859ad9d700 10 monclient(hunting): get_auth_request con 0x7f859c096a50 auth_method 0 2023-09-30T01:54:35.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f859ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:35.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f859ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:35.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.843+0000 7f859ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:35.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.844+0000 7f859a59c700 10 monclient(hunting): get_auth_request con 0x7f859c097360 auth_method 0 2023-09-30T01:54:35.859 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.844+0000 7f859a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:35.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.844+0000 7f859a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:35.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.844+0000 7f859a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:35.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.844+0000 7f859ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:35.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.844+0000 7f859ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:35.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.844+0000 7f859ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:35.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.844+0000 7f859b59e700 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2023-09-30T01:54:35.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.844+0000 7f859b59e700 1 -- 172.21.15.50:0/1491950891 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f859c097360 msgr2=0x7f859c142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:35.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.844+0000 7f859b59e700 1 --2- 172.21.15.50:0/1491950891 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f859c097360 0x7f859c142860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.844+0000 7f859b59e700 1 -- 172.21.15.50:0/1491950891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f859c096a50 msgr2=0x7f859c142320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:35.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.844+0000 7f859b59e700 1 --2- 172.21.15.50:0/1491950891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f859c096a50 0x7f859c142320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.844+0000 7f859b59e700 10 monclient: _finish_hunting 0 2023-09-30T01:54:35.860 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.844+0000 7f859b59e700 1 monclient: found mon.noname-b 2023-09-30T01:54:35.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.844+0000 7f859b59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:35.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.844+0000 7f859b59e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3300/0 2023-09-30T01:54:35.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.844+0000 7f859b59e700 1 -- 172.21.15.50:0/1491950891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8590009420 con 0x7f859c0a2390 2023-09-30T01:54:35.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.844+0000 7f859a59c700 1 --2- 172.21.15.50:0/1491950891 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f859c097360 0x7f859c142860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:54:35.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f859b59e700 1 -- 172.21.15.50:0/1491950891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f859c0a2390 msgr2=0x7f859c09bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:35.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f859b59e700 1 --2- 172.21.15.50:0/1491950891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f859c0a2390 0x7f859c09bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f859400ccc0 tx=0x7f8594005400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:35.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f8582ffd700 1 -- 172.21.15.50:0/1491950891 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859400ab40 con 0x7f859c0a2390 2023-09-30T01:54:35.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f8582ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:35.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f8582ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:54:35.861 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f8582ffd700 10 monclient: dump: 2023-09-30T01:54:35.861 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:35.862 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:35.862 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.862 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.862 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:35.862 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:35.862 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:35.862 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:35.862 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:35.862 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:35.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f8582ffd700 10 monclient: _finish_auth 0 2023-09-30T01:54:35.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f8582ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:35.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f85a115e700 5 monclient: authenticate success, global_id 4120 2023-09-30T01:54:35.862 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f8582ffd700 1 -- 172.21.15.50:0/1491950891 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f859400acc0 con 0x7f859c0a2390 2023-09-30T01:54:35.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f8582ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:35.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f8582ffd700 1 -- 172.21.15.50:0/1491950891 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8594007340 con 0x7f859c0a2390 2023-09-30T01:54:35.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f8582ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:35.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f8582ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:54:35.863 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f8582ffd700 10 monclient: dump: 2023-09-30T01:54:35.863 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:35.863 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:35.863 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.863 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:35.863 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:35.863 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:35.863 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:35.864 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:35.864 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:35.864 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:35.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f85a115e700 10 monclient: _renew_subs 2023-09-30T01:54:35.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f859959a700 4 set_mon_vals no callback set 2023-09-30T01:54:35.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f85a115e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:35.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f85a115e700 1 -- 172.21.15.50:0/1491950891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f859c09f160 con 0x7f859c0a2390 2023-09-30T01:54:35.864 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f85a115e700 10 monclient: _renew_subs 2023-09-30T01:54:35.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f85a115e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:35.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f85a115e700 1 -- 172.21.15.50:0/1491950891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f859c142da0 con 0x7f859c0a2390 2023-09-30T01:54:35.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f8582ffd700 1 -- 172.21.15.50:0/1491950891 <== mon.1 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f85940076f0 con 0x7f859c0a2390 2023-09-30T01:54:35.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.846+0000 7f8582ffd700 1 -- 172.21.15.50:0/1491950891 <== mon.1 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7f859401b9c0 con 0x7f859c0a2390 2023-09-30T01:54:35.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.847+0000 7f85a115e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:35.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.847+0000 7f85a115e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:35.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.847+0000 7f85a115e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:35.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.847+0000 7f85a115e700 1 -- 172.21.15.50:0/1491950891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f859c0a4be0 con 0x7f859c0a2390 2023-09-30T01:54:35.865 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.849+0000 7f8582ffd700 1 -- 172.21.15.50:0/1491950891 <== mon.1 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f8594007ae0 con 0x7f859c0a2390 2023-09-30T01:54:35.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.849+0000 7f8582ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:35.866 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.849+0000 7f8582ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:35.923 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:35.923+0000 7fc92c365180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T01:54:35.933 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.933+0000 7f85a115e700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b39c552e-f5cc-4e4f-90a3-5aa5f5127a27", "id": 2}] 2023-09-30T01:54:35.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.933+0000 7f85a115e700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b39c552e-f5cc-4e4f-90a3-5aa5f5127a27", "id": 2}] 2023-09-30T01:54:35.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.933+0000 7f85a115e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:35.934 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.933+0000 7f85a115e700 1 -- 172.21.15.50:0/1491950891 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b39c552e-f5cc-4e4f-90a3-5aa5f5127a27", "id": 2} v 0) v1 -- 0x7f859c0a4900 con 0x7f859c0a2390 2023-09-30T01:54:35.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.936+0000 7f8582ffd700 1 -- 172.21.15.50:0/1491950891 <== mon.1 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b39c552e-f5cc-4e4f-90a3-5aa5f5127a27", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f859402a0b0 con 0x7f859c0a2390 2023-09-30T01:54:35.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.936+0000 7f8582ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b39c552e-f5cc-4e4f-90a3-5aa5f5127a27", "id": 2}] 2023-09-30T01:54:35.936 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.936+0000 7f8582ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:35.936 INFO:teuthology.orchestra.run.smithi050.stdout:2 2023-09-30T01:54:35.937 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.937+0000 7f85a115e700 10 monclient: shutdown 2023-09-30T01:54:35.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.937+0000 7f85a115e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:35.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.937+0000 7f85a115e700 1 -- 172.21.15.50:0/1491950891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f859c0a2390 msgr2=0x7f859c09bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:35.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.937+0000 7f85a115e700 1 --2- 172.21.15.50:0/1491950891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f859c0a2390 0x7f859c09bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f859400ccc0 tx=0x7f8594005400 comp rx=0 tx=0).stop 2023-09-30T01:54:35.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.937+0000 7f85a115e700 1 -- 172.21.15.50:0/1491950891 shutdown_connections 2023-09-30T01:54:35.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.937+0000 7f85a115e700 1 --2- 172.21.15.50:0/1491950891 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f859c096a50 0x7f859c142320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.937+0000 7f85a115e700 1 --2- 172.21.15.50:0/1491950891 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f859c0a2390 0x7f859c09bb70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.938 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.937+0000 7f85a115e700 1 --2- 172.21.15.50:0/1491950891 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f859c097360 0x7f859c142860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:35.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.937+0000 7f85a115e700 1 -- 172.21.15.50:0/1491950891 >> 172.21.15.50:0/1491950891 conn(0x7f859c070cf0 msgr2=0x7f859c133b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:35.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.937+0000 7f85a115e700 1 -- 172.21.15.50:0/1491950891 shutdown_connections 2023-09-30T01:54:35.939 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:35.937+0000 7f85a115e700 1 -- 172.21.15.50:0/1491950891 wait complete. 2023-09-30T01:54:35.949 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd new 15cd89ae-049a-4e1a-8c6b-7d086ca6e1df 3 2023-09-30T01:54:36.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.067+0000 7f5701c78700 10 monclient: get_monmap_and_config 2023-09-30T01:54:36.067 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.067+0000 7f5701c78700 10 monclient: build_initial_monmap 2023-09-30T01:54:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.067+0000 7f5701c78700 10 monclient: monmap: 2023-09-30T01:54:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-09-30T01:54:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:36.068585+0000 2023-09-30T01:54:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:36.068585+0000 2023-09-30T01:54:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:36.068 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:36.069 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:36.069 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.068+0000 7f5701c78700 1 Processor -- start 2023-09-30T01:54:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.068+0000 7f5701c78700 1 -- start start 2023-09-30T01:54:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.068+0000 7f5701c78700 10 monclient: init 2023-09-30T01:54:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.068+0000 7f5701c78700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.068+0000 7f5701c78700 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-09-30T01:54:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.068+0000 7f5701c78700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f56fc12ff60 addr v1:172.21.15.50:6789/0 2023-09-30T01:54:36.073 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.068+0000 7f5701c78700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f56fc130ab0 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.068+0000 7f5701c78700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f56fc098c10 0x7f56fc1327e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.068+0000 7f5701c78700 10 monclient(hunting): picked mon.noname-c con 0x7f56fc098c10 addr v2:172.21.15.50:3300/0 2023-09-30T01:54:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.068+0000 7f5701c78700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56fc096ac0 con 0x7f56fc130ab0 2023-09-30T01:54:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.068+0000 7f5701c78700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56fc130330 con 0x7f56fc12ff60 2023-09-30T01:54:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.068+0000 7f5701c78700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.068+0000 7f5701c78700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56fc096cf0 con 0x7f56fc098c10 2023-09-30T01:54:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.068+0000 7f5701c78700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.068+0000 7f5701c78700 10 monclient(hunting): authenticate will time out at 1166.296973s 2023-09-30T01:54:36.074 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.069+0000 7f5701477700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f56fc098c10 0x7f56fc1327e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.069+0000 7f5700c76700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f56fc12ff60 0x7f56fc098500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:45388/0 (socket says 172.21.15.50:45388) 2023-09-30T01:54:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.069+0000 7f5700c76700 1 -- 172.21.15.50:0/2680363551 learned_addr learned my addr 172.21.15.50:0/2680363551 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-09-30T01:54:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.069+0000 7f5701477700 10 monclient(hunting): get_auth_request con 0x7f56fc098c10 auth_method 0 2023-09-30T01:54:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.069+0000 7f5701477700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.069+0000 7f5701477700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.069+0000 7f5701477700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.069+0000 7f5701477700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.069+0000 7f5701477700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.069+0000 7f5701477700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.069+0000 7f5701477700 10 monclient(hunting): handle_auth_done global_id 4123 payload 274 2023-09-30T01:54:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.069+0000 7f5701477700 1 -- 172.21.15.50:0/2680363551 >> v1:172.21.15.50:6789/0 conn(0x7f56fc12ff60 legacy=0x7f56fc098500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.069+0000 7f5701477700 1 -- 172.21.15.50:0/2680363551 >> v1:172.21.15.7:6789/0 conn(0x7f56fc130ab0 legacy=0x7f56fc1320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.069+0000 7f5701477700 10 monclient: _finish_hunting 0 2023-09-30T01:54:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.069+0000 7f5701477700 1 monclient: found mon.noname-c 2023-09-30T01:54:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.069+0000 7f5701477700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:36.076 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.069+0000 7f5701477700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3300/0 2023-09-30T01:54:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.069+0000 7f5701477700 1 -- 172.21.15.50:0/2680363551 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56fc0a07b0 con 0x7f56fc098c10 2023-09-30T01:54:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.071+0000 7f5701477700 1 -- 172.21.15.50:0/2680363551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56fc098c10 msgr2=0x7f56fc1327e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.071+0000 7f5701477700 1 --2- 172.21.15.50:0/2680363551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56fc098c10 0x7f56fc1327e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f56f4009e60 tx=0x7f56f4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5eb2d4cee2d4da5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.071+0000 7f56fb7fe700 1 -- 172.21.15.50:0/2680363551 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f400e070 con 0x7f56fc098c10 2023-09-30T01:54:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.071+0000 7f56fb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.071+0000 7f56fb7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.071+0000 7f56fb7fe700 10 monclient: dump: 2023-09-30T01:54:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.077 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:36.078 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.071+0000 7f56fb7fe700 10 monclient: _finish_auth 0 2023-09-30T01:54:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.071+0000 7f56fb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.071+0000 7f56fb7fe700 1 -- 172.21.15.50:0/2680363551 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f4012430 con 0x7f56fc098c10 2023-09-30T01:54:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.071+0000 7f5701c78700 5 monclient: authenticate success, global_id 4123 2023-09-30T01:54:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.071+0000 7f5701c78700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.071+0000 7f56fb7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:36.078 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.071+0000 7f56fb7fe700 1 -- 172.21.15.50:0/2680363551 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f400bea0 con 0x7f56fc098c10 2023-09-30T01:54:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.071+0000 7f5701c78700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.071+0000 7f5701c78700 4 set_mon_vals no callback set 2023-09-30T01:54:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.071+0000 7f56fb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.071+0000 7f56fb7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:54:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.071+0000 7f56fb7fe700 10 monclient: dump: 2023-09-30T01:54:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:36.079 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:36.080 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.072+0000 7f5701c78700 10 monclient: shutdown 2023-09-30T01:54:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.072+0000 7f5701c78700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.072+0000 7f5701c78700 1 -- 172.21.15.50:0/2680363551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56fc098c10 msgr2=0x7f56fc1327e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.072+0000 7f5701c78700 1 --2- 172.21.15.50:0/2680363551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56fc098c10 0x7f56fc1327e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f56f4009e60 tx=0x7f56f4009780 comp rx=0 tx=0).stop 2023-09-30T01:54:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.072+0000 7f5701c78700 1 -- 172.21.15.50:0/2680363551 shutdown_connections 2023-09-30T01:54:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.072+0000 7f5701c78700 1 --2- 172.21.15.50:0/2680363551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56fc098c10 0x7f56fc1327e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.072+0000 7f5701c78700 1 -- 172.21.15.50:0/2680363551 >> 172.21.15.50:0/2680363551 conn(0x7f56fc071870 msgr2=0x7f56fc0a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:36.080 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.072+0000 7f5701c78700 1 -- 172.21.15.50:0/2680363551 shutdown_connections 2023-09-30T01:54:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.072+0000 7f5701c78700 1 -- 172.21.15.50:0/2680363551 wait complete. 2023-09-30T01:54:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.073+0000 7f5701c78700 10 monclient: build_initial_monmap 2023-09-30T01:54:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.073+0000 7f5701c78700 10 monclient: monmap: 2023-09-30T01:54:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-09-30T01:54:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-09-30T01:54:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-09-30T01:54:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:36.081 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:36.082 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:36.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.073+0000 7f5701c78700 1 Processor -- start 2023-09-30T01:54:36.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.073+0000 7f5701c78700 1 -- start start 2023-09-30T01:54:36.082 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.073+0000 7f5701c78700 10 monclient: init 2023-09-30T01:54:36.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5701c78700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:36.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5701c78700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:54:36.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5701c78700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56fc098c10 0x7f56fc1414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:36.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5701c78700 10 monclient(hunting): picked mon.noname-a con 0x7f56fc098c10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:36.086 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5701c78700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56fc12ff60 0x7f56fc12f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:36.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5701c78700 10 monclient(hunting): picked mon.noname-b con 0x7f56fc12ff60 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:36.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5701c78700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f56fc130ab0 0x7f56fc12f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:36.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5701c78700 10 monclient(hunting): picked mon.noname-c con 0x7f56fc130ab0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:36.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5701c78700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:36.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5701c78700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56fc06b020 con 0x7f56fc098c10 2023-09-30T01:54:36.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5701c78700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:36.087 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5701c78700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56fc09e950 con 0x7f56fc12ff60 2023-09-30T01:54:36.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5701c78700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:36.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5701c78700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56fc142370 con 0x7f56fc130ab0 2023-09-30T01:54:36.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5701c78700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:36.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5701c78700 10 monclient(hunting): authenticate will time out at 1166.302120s 2023-09-30T01:54:36.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f56fbfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56fc12ff60 0x7f56fc12f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:36.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f56fbfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56fc12ff60 0x7f56fc12f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59724/0 (socket says 172.21.15.50:59724) 2023-09-30T01:54:36.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f56fbfff700 1 -- 172.21.15.50:0/218283239 learned_addr learned my addr 172.21.15.50:0/218283239 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-09-30T01:54:36.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f56fbfff700 10 monclient(hunting): get_auth_request con 0x7f56fc12ff60 auth_method 0 2023-09-30T01:54:36.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f56fbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:36.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f56fbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:36.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f56fbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:36.088 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f56fbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:36.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f56fbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:36.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f56fbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:36.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5700c76700 1 --2- 172.21.15.50:0/218283239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56fc098c10 0x7f56fc1414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:36.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5700c76700 10 monclient(hunting): get_auth_request con 0x7f56fc098c10 auth_method 0 2023-09-30T01:54:36.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5700c76700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:36.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5700c76700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:36.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5701477700 1 --2- 172.21.15.50:0/218283239 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f56fc130ab0 0x7f56fc12f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:36.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5700c76700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:36.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5701477700 10 monclient(hunting): get_auth_request con 0x7f56fc130ab0 auth_method 0 2023-09-30T01:54:36.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5701477700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:36.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5701477700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:36.089 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5701477700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:36.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f56fbfff700 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2023-09-30T01:54:36.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f56fbfff700 1 -- 172.21.15.50:0/218283239 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f56fc130ab0 msgr2=0x7f56fc12f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:36.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f56fbfff700 1 --2- 172.21.15.50:0/218283239 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f56fc130ab0 0x7f56fc12f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f56fbfff700 1 -- 172.21.15.50:0/218283239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56fc098c10 msgr2=0x7f56fc1414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:36.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f56fbfff700 1 --2- 172.21.15.50:0/218283239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56fc098c10 0x7f56fc1414e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f56fbfff700 10 monclient: _finish_hunting 0 2023-09-30T01:54:36.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f56fbfff700 1 monclient: found mon.noname-b 2023-09-30T01:54:36.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f56fbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:36.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f56fbfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3300/0 2023-09-30T01:54:36.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f56fbfff700 1 -- 172.21.15.50:0/218283239 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f4009a20 con 0x7f56fc12ff60 2023-09-30T01:54:36.090 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.074+0000 7f5700c76700 1 --2- 172.21.15.50:0/218283239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56fc098c10 0x7f56fc1414e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:54:36.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.076+0000 7f56fbfff700 1 -- 172.21.15.50:0/218283239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56fc12ff60 msgr2=0x7f56fc12f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:36.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.076+0000 7f56fbfff700 1 --2- 172.21.15.50:0/218283239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56fc12ff60 0x7f56fc12f1a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f56e800c5d0 tx=0x7f56e8007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:36.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.076+0000 7f56f8ff9700 1 -- 172.21.15.50:0/218283239 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e80117a0 con 0x7f56fc12ff60 2023-09-30T01:54:36.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.076+0000 7f56f8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:36.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.076+0000 7f56f8ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:54:36.091 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.076+0000 7f56f8ff9700 10 monclient: dump: 2023-09-30T01:54:36.091 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:36.091 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:36.091 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.091 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.091 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:36.091 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:36.092 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.076+0000 7f56f8ff9700 10 monclient: _finish_auth 0 2023-09-30T01:54:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.076+0000 7f56f8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.076+0000 7f5701c78700 5 monclient: authenticate success, global_id 4126 2023-09-30T01:54:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.076+0000 7f56f8ff9700 1 -- 172.21.15.50:0/218283239 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e8011900 con 0x7f56fc12ff60 2023-09-30T01:54:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.076+0000 7f56f8ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.076+0000 7f5701c78700 10 monclient: _renew_subs 2023-09-30T01:54:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.076+0000 7f56f8ff9700 1 -- 172.21.15.50:0/218283239 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e8011cc0 con 0x7f56fc12ff60 2023-09-30T01:54:36.092 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.076+0000 7f5701c78700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:36.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.077+0000 7f5701c78700 1 -- 172.21.15.50:0/218283239 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56fc121310 con 0x7f56fc12ff60 2023-09-30T01:54:36.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.077+0000 7f56f8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:36.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.077+0000 7f56f8ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:54:36.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.076+0000 7f56faffd700 4 set_mon_vals no callback set 2023-09-30T01:54:36.093 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.077+0000 7f56f8ff9700 10 monclient: dump: 2023-09-30T01:54:36.093 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:36.093 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:36.093 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.093 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.093 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:36.093 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:36.093 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:36.094 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:36.094 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:36.094 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:36.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.077+0000 7f5701c78700 10 monclient: _renew_subs 2023-09-30T01:54:36.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.077+0000 7f5701c78700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:36.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.077+0000 7f5701c78700 1 -- 172.21.15.50:0/218283239 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56fc120420 con 0x7f56fc12ff60 2023-09-30T01:54:36.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.077+0000 7f56f8ff9700 1 -- 172.21.15.50:0/218283239 <== mon.1 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f56e8012630 con 0x7f56fc12ff60 2023-09-30T01:54:36.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.077+0000 7f56f8ff9700 1 -- 172.21.15.50:0/218283239 <== mon.1 v2:172.21.15.50:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f56e801c900 con 0x7f56fc12ff60 2023-09-30T01:54:36.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.078+0000 7f5701c78700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:36.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.078+0000 7f5701c78700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:36.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.078+0000 7f5701c78700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:36.094 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.078+0000 7f5701c78700 1 -- 172.21.15.50:0/218283239 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56fc064c10 con 0x7f56fc12ff60 2023-09-30T01:54:36.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.081+0000 7f56f8ff9700 1 -- 172.21.15.50:0/218283239 <== mon.1 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f56e8012ac0 con 0x7f56fc12ff60 2023-09-30T01:54:36.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.081+0000 7f56f8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:36.095 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.081+0000 7f56f8ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:36.122 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:36.122+0000 7fc92c365180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T01:54:36.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.165+0000 7f5701c78700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "15cd89ae-049a-4e1a-8c6b-7d086ca6e1df", "id": 3}] 2023-09-30T01:54:36.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.165+0000 7f5701c78700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "15cd89ae-049a-4e1a-8c6b-7d086ca6e1df", "id": 3}] 2023-09-30T01:54:36.165 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.165+0000 7f5701c78700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:36.166 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.165+0000 7f5701c78700 1 -- 172.21.15.50:0/218283239 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd new", "uuid": "15cd89ae-049a-4e1a-8c6b-7d086ca6e1df", "id": 3} v 0) v1 -- 0x7f56fc0a1b20 con 0x7f56fc12ff60 2023-09-30T01:54:36.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.169+0000 7f56f8ff9700 1 -- 172.21.15.50:0/218283239 <== mon.1 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "15cd89ae-049a-4e1a-8c6b-7d086ca6e1df", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f56e8012d20 con 0x7f56fc12ff60 2023-09-30T01:54:36.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.169+0000 7f56f8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "15cd89ae-049a-4e1a-8c6b-7d086ca6e1df", "id": 3}] 2023-09-30T01:54:36.169 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.169+0000 7f56f8ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:36.170 INFO:teuthology.orchestra.run.smithi050.stdout:3 2023-09-30T01:54:36.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.170+0000 7f5701c78700 10 monclient: shutdown 2023-09-30T01:54:36.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.170+0000 7f5701c78700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:36.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.170+0000 7f5701c78700 1 -- 172.21.15.50:0/218283239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56fc12ff60 msgr2=0x7f56fc12f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:36.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.170+0000 7f5701c78700 1 --2- 172.21.15.50:0/218283239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56fc12ff60 0x7f56fc12f1a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f56e800c5d0 tx=0x7f56e8007f60 comp rx=0 tx=0).stop 2023-09-30T01:54:36.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.170+0000 7f5701c78700 1 -- 172.21.15.50:0/218283239 shutdown_connections 2023-09-30T01:54:36.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.170+0000 7f5701c78700 1 --2- 172.21.15.50:0/218283239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56fc098c10 0x7f56fc1414e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.170+0000 7f5701c78700 1 --2- 172.21.15.50:0/218283239 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f56fc130ab0 0x7f56fc12f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.170+0000 7f5701c78700 1 --2- 172.21.15.50:0/218283239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56fc12ff60 0x7f56fc12f1a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.170+0000 7f5701c78700 1 -- 172.21.15.50:0/218283239 >> 172.21.15.50:0/218283239 conn(0x7f56fc071870 msgr2=0x7f56fc09df20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:36.171 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.170+0000 7f5701c78700 1 -- 172.21.15.50:0/218283239 shutdown_connections 2023-09-30T01:54:36.172 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.170+0000 7f5701c78700 1 -- 172.21.15.50:0/218283239 wait complete. 2023-09-30T01:54:36.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd new de6773bd-d286-4ab9-b91f-8a90ee3073d2 4 2023-09-30T01:54:36.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.299+0000 7f1228930700 10 monclient: get_monmap_and_config 2023-09-30T01:54:36.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.299+0000 7f1228930700 10 monclient: build_initial_monmap 2023-09-30T01:54:36.299 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.299+0000 7f1228930700 10 monclient: monmap: 2023-09-30T01:54:36.300 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-09-30T01:54:36.300 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:36.300 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:36.300454+0000 2023-09-30T01:54:36.300 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:36.300454+0000 2023-09-30T01:54:36.300 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:36.300 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:36.300 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:36.300 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:36.300 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:36.300 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:36.300 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:36.300 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:36.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.300+0000 7f1228930700 1 Processor -- start 2023-09-30T01:54:36.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.300+0000 7f1228930700 1 -- start start 2023-09-30T01:54:36.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.300+0000 7f1228930700 10 monclient: init 2023-09-30T01:54:36.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.300+0000 7f1228930700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:36.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.300+0000 7f1228930700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-09-30T01:54:36.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.300+0000 7f1228930700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1224096a50 0x7f12241327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:36.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.300+0000 7f1228930700 10 monclient(hunting): picked mon.noname-b con 0x7f1224096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:36.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.300+0000 7f1228930700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12240a2390 addr v1:172.21.15.50:6789/0 2023-09-30T01:54:36.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.300+0000 7f1228930700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1224097530 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:36.303 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.300+0000 7f1228930700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12240980e0 con 0x7f1224097530 2023-09-30T01:54:36.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.300+0000 7f1228930700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12240991a0 con 0x7f12240a2390 2023-09-30T01:54:36.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.300+0000 7f1228930700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:36.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.300+0000 7f1228930700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1224099480 con 0x7f1224096a50 2023-09-30T01:54:36.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.300+0000 7f1228930700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:36.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.300+0000 7f1228930700 10 monclient(hunting): authenticate will time out at 1166.528947s 2023-09-30T01:54:36.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.301+0000 7f1221d9b700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f12240a2390 0x7f1224096e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:45404/0 (socket says 172.21.15.50:45404) 2023-09-30T01:54:36.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.301+0000 7f1221d9b700 1 -- 172.21.15.50:0/3944677291 learned_addr learned my addr 172.21.15.50:0/3944677291 (peer_addr_for_me v1:172.21.15.50:0/0) 2023-09-30T01:54:36.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.301+0000 7f122259c700 1 --2- 172.21.15.50:0/3944677291 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1224096a50 0x7f12241327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:36.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.301+0000 7f122259c700 10 monclient(hunting): get_auth_request con 0x7f1224096a50 auth_method 0 2023-09-30T01:54:36.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.301+0000 7f122259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:36.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.301+0000 7f122259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:36.304 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.301+0000 7f122259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:36.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.301+0000 7f122259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:36.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.301+0000 7f122259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:36.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.301+0000 7f122259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:36.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.301+0000 7f122259c700 10 monclient(hunting): handle_auth_done global_id 4124 payload 274 2023-09-30T01:54:36.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.301+0000 7f122259c700 1 -- 172.21.15.50:0/3944677291 >> v1:172.21.15.50:6789/0 conn(0x7f12240a2390 legacy=0x7f1224096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:36.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.301+0000 7f122259c700 1 -- 172.21.15.50:0/3944677291 >> v1:172.21.15.7:6789/0 conn(0x7f1224097530 legacy=0x7f1224139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:36.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.301+0000 7f122259c700 10 monclient: _finish_hunting 0 2023-09-30T01:54:36.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.301+0000 7f122259c700 1 monclient: found mon.noname-b 2023-09-30T01:54:36.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.301+0000 7f122259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:36.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.301+0000 7f122259c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:54:36.305 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.301+0000 7f122259c700 1 -- 172.21.15.50:0/3944677291 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1224097bf0 con 0x7f1224096a50 2023-09-30T01:54:36.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.303+0000 7f122259c700 1 -- 172.21.15.50:0/3944677291 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1224096a50 msgr2=0x7f12241327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:36.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.303+0000 7f122259c700 1 --2- 172.21.15.50:0/3944677291 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1224096a50 0x7f12241327a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f120c009910 tx=0x7f120c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ae3b128897ab581 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:36.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f122159a700 1 -- 172.21.15.50:0/3944677291 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120c00ba30 con 0x7f1224096a50 2023-09-30T01:54:36.311 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f122159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f122159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:54:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f122159a700 10 monclient: dump: 2023-09-30T01:54:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:36.313 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f122159a700 10 monclient: _finish_auth 0 2023-09-30T01:54:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f122159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f122159a700 1 -- 172.21.15.50:0/3944677291 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120c00bb90 con 0x7f1224096a50 2023-09-30T01:54:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f1228930700 5 monclient: authenticate success, global_id 4124 2023-09-30T01:54:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f1228930700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f122159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f1228930700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f122159a700 1 -- 172.21.15.50:0/3944677291 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120c011420 con 0x7f1224096a50 2023-09-30T01:54:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f122159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:36.313 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f1228930700 4 set_mon_vals no callback set 2023-09-30T01:54:36.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f122159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:36.314 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f122159a700 10 monclient: dump: 2023-09-30T01:54:36.314 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:36.314 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:36.314 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.314 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.314 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:36.314 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:36.314 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:36.314 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:36.314 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:36.315 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:36.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f1228930700 10 monclient: shutdown 2023-09-30T01:54:36.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f1228930700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:36.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f1228930700 1 -- 172.21.15.50:0/3944677291 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1224096a50 msgr2=0x7f12241327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:36.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f1228930700 1 --2- 172.21.15.50:0/3944677291 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1224096a50 0x7f12241327a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f120c009910 tx=0x7f120c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f1228930700 1 -- 172.21.15.50:0/3944677291 shutdown_connections 2023-09-30T01:54:36.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f1228930700 1 --2- 172.21.15.50:0/3944677291 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1224096a50 0x7f12241327a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f1228930700 1 -- 172.21.15.50:0/3944677291 >> 172.21.15.50:0/3944677291 conn(0x7f1224070cf0 msgr2=0x7f12240710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:36.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f1228930700 1 -- 172.21.15.50:0/3944677291 shutdown_connections 2023-09-30T01:54:36.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.304+0000 7f1228930700 1 -- 172.21.15.50:0/3944677291 wait complete. 2023-09-30T01:54:36.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.305+0000 7f1228930700 10 monclient: build_initial_monmap 2023-09-30T01:54:36.315 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.305+0000 7f1228930700 10 monclient: monmap: 2023-09-30T01:54:36.316 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-09-30T01:54:36.316 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:36.316 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-09-30T01:54:36.316 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-09-30T01:54:36.316 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:36.316 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:36.316 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:36.316 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:36.316 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:36.316 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:36.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.305+0000 7f1228930700 1 Processor -- start 2023-09-30T01:54:36.316 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.305+0000 7f1228930700 1 -- start start 2023-09-30T01:54:36.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.305+0000 7f1228930700 10 monclient: init 2023-09-30T01:54:36.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1228930700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:36.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1228930700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:54:36.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1228930700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1224096a50 0x7f122409bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:36.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1228930700 10 monclient(hunting): picked mon.noname-c con 0x7f1224096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:36.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1228930700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1224097530 0x7f122409c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:36.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1228930700 10 monclient(hunting): picked mon.noname-b con 0x7f1224097530 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:36.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1228930700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12240a2390 0x7f122409c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:36.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1228930700 10 monclient(hunting): picked mon.noname-a con 0x7f12240a2390 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:36.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1228930700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:36.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1228930700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122409d5f0 con 0x7f12240a2390 2023-09-30T01:54:36.317 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1228930700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:36.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1228930700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122413abc0 con 0x7f1224097530 2023-09-30T01:54:36.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1228930700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:36.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1228930700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f122409f300 con 0x7f1224096a50 2023-09-30T01:54:36.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1228930700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:36.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1228930700 10 monclient(hunting): authenticate will time out at 1166.534323s 2023-09-30T01:54:36.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1221d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1224097530 0x7f122409c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:36.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1221d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1224097530 0x7f122409c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59726/0 (socket says 172.21.15.50:59726) 2023-09-30T01:54:36.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f122259c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1224096a50 0x7f122409bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:36.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1221d9b700 1 -- 172.21.15.50:0/3320979602 learned_addr learned my addr 172.21.15.50:0/3320979602 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-09-30T01:54:36.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1221d9b700 10 monclient(hunting): get_auth_request con 0x7f1224097530 auth_method 0 2023-09-30T01:54:36.318 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1221d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:36.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1221d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:36.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1221d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:36.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f122259c700 10 monclient(hunting): get_auth_request con 0x7f1224096a50 auth_method 0 2023-09-30T01:54:36.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f122259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:36.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f122259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:36.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f122259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:36.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1222d9d700 1 --2- 172.21.15.50:0/3320979602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12240a2390 0x7f122409c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:36.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1222d9d700 10 monclient(hunting): get_auth_request con 0x7f12240a2390 auth_method 0 2023-09-30T01:54:36.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1222d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:36.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1222d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:36.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1222d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:36.319 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1221d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:36.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1221d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:36.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1221d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:36.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f122259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:36.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f122259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:36.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f122259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:36.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1222d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:36.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1222d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:36.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1222d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:36.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1221d9b700 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2023-09-30T01:54:36.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1221d9b700 1 -- 172.21.15.50:0/3320979602 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1224096a50 msgr2=0x7f122409bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:36.320 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1221d9b700 1 --2- 172.21.15.50:0/3320979602 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1224096a50 0x7f122409bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1221d9b700 1 -- 172.21.15.50:0/3320979602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12240a2390 msgr2=0x7f122409c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:36.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1221d9b700 1 --2- 172.21.15.50:0/3320979602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12240a2390 0x7f122409c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1221d9b700 10 monclient: _finish_hunting 0 2023-09-30T01:54:36.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1221d9b700 1 monclient: found mon.noname-b 2023-09-30T01:54:36.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1221d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:36.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1221d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3300/0 2023-09-30T01:54:36.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.306+0000 7f1221d9b700 1 -- 172.21.15.50:0/3320979602 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120c009420 con 0x7f1224097530 2023-09-30T01:54:36.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f1221d9b700 1 -- 172.21.15.50:0/3320979602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1224097530 msgr2=0x7f122409c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:36.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f1221d9b700 1 --2- 172.21.15.50:0/3320979602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1224097530 0x7f122409c1c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f121400c610 tx=0x7f1214004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:36.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f121a7fc700 1 -- 172.21.15.50:0/3320979602 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1214005d40 con 0x7f1224097530 2023-09-30T01:54:36.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f121a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:36.321 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f121a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:54:36.322 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f121a7fc700 10 monclient: dump: 2023-09-30T01:54:36.322 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:36.322 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:36.322 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.322 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.322 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:36.322 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:36.322 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:36.322 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:36.322 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:36.323 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:36.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f121a7fc700 10 monclient: _finish_auth 0 2023-09-30T01:54:36.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f121a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:36.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f1228930700 5 monclient: authenticate success, global_id 4129 2023-09-30T01:54:36.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f121a7fc700 1 -- 172.21.15.50:0/3320979602 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12140070d0 con 0x7f1224097530 2023-09-30T01:54:36.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f121a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:36.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f1228930700 10 monclient: _renew_subs 2023-09-30T01:54:36.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f121a7fc700 1 -- 172.21.15.50:0/3320979602 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1214007490 con 0x7f1224097530 2023-09-30T01:54:36.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f1228930700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:36.323 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f1228930700 1 -- 172.21.15.50:0/3320979602 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12240a0230 con 0x7f1224097530 2023-09-30T01:54:36.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f121a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:36.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f121a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:54:36.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f1220d99700 4 set_mon_vals no callback set 2023-09-30T01:54:36.324 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f121a7fc700 10 monclient: dump: 2023-09-30T01:54:36.324 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:36.324 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:36.324 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.324 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.324 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:36.325 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:36.325 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:36.325 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:36.325 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:36.325 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:36.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f1228930700 10 monclient: _renew_subs 2023-09-30T01:54:36.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f1228930700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:36.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f1228930700 1 -- 172.21.15.50:0/3320979602 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f122409ec70 con 0x7f1224097530 2023-09-30T01:54:36.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f121a7fc700 1 -- 172.21.15.50:0/3320979602 <== mon.1 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f121401c6c0 con 0x7f1224097530 2023-09-30T01:54:36.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.309+0000 7f121a7fc700 1 -- 172.21.15.50:0/3320979602 <== mon.1 v2:172.21.15.50:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f1214012910 con 0x7f1224097530 2023-09-30T01:54:36.325 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.310+0000 7f1228930700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:36.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.310+0000 7f1228930700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:36.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.310+0000 7f1228930700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:36.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.310+0000 7f1228930700 1 -- 172.21.15.50:0/3320979602 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1224097150 con 0x7f1224097530 2023-09-30T01:54:36.326 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:36.310+0000 7fc92c365180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T01:54:36.326 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.312+0000 7f121a7fc700 1 -- 172.21.15.50:0/3320979602 <== mon.1 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f12240a0230 con 0x7f1224097530 2023-09-30T01:54:36.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.312+0000 7f121a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:36.327 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.312+0000 7f121a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:36.392 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:36.392+0000 7fc92c365180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T01:54:36.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.395+0000 7f1228930700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "de6773bd-d286-4ab9-b91f-8a90ee3073d2", "id": 4}] 2023-09-30T01:54:36.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.395+0000 7f1228930700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "de6773bd-d286-4ab9-b91f-8a90ee3073d2", "id": 4}] 2023-09-30T01:54:36.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.395+0000 7f1228930700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:36.395 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.395+0000 7f1228930700 1 -- 172.21.15.50:0/3320979602 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd new", "uuid": "de6773bd-d286-4ab9-b91f-8a90ee3073d2", "id": 4} v 0) v1 -- 0x7f122409f440 con 0x7f1224097530 2023-09-30T01:54:36.398 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.398+0000 7f121a7fc700 1 -- 172.21.15.50:0/3320979602 <== mon.1 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "de6773bd-d286-4ab9-b91f-8a90ee3073d2", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f122409f440 con 0x7f1224097530 2023-09-30T01:54:36.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.398+0000 7f121a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "de6773bd-d286-4ab9-b91f-8a90ee3073d2", "id": 4}] 2023-09-30T01:54:36.399 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.398+0000 7f121a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:36.399 INFO:teuthology.orchestra.run.smithi050.stdout:4 2023-09-30T01:54:36.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.399+0000 7f1228930700 10 monclient: shutdown 2023-09-30T01:54:36.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.399+0000 7f1228930700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:36.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.399+0000 7f1228930700 1 -- 172.21.15.50:0/3320979602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1224097530 msgr2=0x7f122409c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:36.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.399+0000 7f1228930700 1 --2- 172.21.15.50:0/3320979602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1224097530 0x7f122409c1c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f121400c610 tx=0x7f1214004a80 comp rx=0 tx=0).stop 2023-09-30T01:54:36.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.399+0000 7f1228930700 1 -- 172.21.15.50:0/3320979602 shutdown_connections 2023-09-30T01:54:36.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.399+0000 7f1228930700 1 --2- 172.21.15.50:0/3320979602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f12240a2390 0x7f122409c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.399+0000 7f1228930700 1 --2- 172.21.15.50:0/3320979602 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1224096a50 0x7f122409bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.400 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.399+0000 7f1228930700 1 --2- 172.21.15.50:0/3320979602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1224097530 0x7f122409c1c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.399+0000 7f1228930700 1 -- 172.21.15.50:0/3320979602 >> 172.21.15.50:0/3320979602 conn(0x7f1224070cf0 msgr2=0x7f12240a4500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:36.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.399+0000 7f1228930700 1 -- 172.21.15.50:0/3320979602 shutdown_connections 2023-09-30T01:54:36.401 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.399+0000 7f1228930700 1 -- 172.21.15.50:0/3320979602 wait complete. 2023-09-30T01:54:36.411 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd new e6a847ab-276e-4d54-9a8f-6df5b8ab54cc 5 2023-09-30T01:54:36.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.531+0000 7fb011185700 10 monclient: get_monmap_and_config 2023-09-30T01:54:36.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.531+0000 7fb011185700 10 monclient: build_initial_monmap 2023-09-30T01:54:36.539 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.531+0000 7fb011185700 10 monclient: monmap: 2023-09-30T01:54:36.539 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-09-30T01:54:36.540 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:36.540 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:36.531885+0000 2023-09-30T01:54:36.540 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:36.531885+0000 2023-09-30T01:54:36.540 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:36.540 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:36.540 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:36.541 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:36.541 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:36.541 INFO:teuthology.orchestra.run.smithi050.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:36.541 INFO:teuthology.orchestra.run.smithi050.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:36.541 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:36.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.531+0000 7fb011185700 1 Processor -- start 2023-09-30T01:54:36.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.531+0000 7fb011185700 1 -- start start 2023-09-30T01:54:36.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.531+0000 7fb011185700 10 monclient: init 2023-09-30T01:54:36.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb011185700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:36.542 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb011185700 10 monclient: _add_conns ranks=[4,2,0,1,3] 2023-09-30T01:54:36.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb011185700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb00c096a50 0x7fb00c1327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:36.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb011185700 10 monclient(hunting): picked mon.noname-c con 0x7fb00c096a50 addr v2:172.21.15.50:3300/0 2023-09-30T01:54:36.543 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb011185700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb00c0a2390 0x7fb00c096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:36.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb011185700 10 monclient(hunting): picked mon.noname-b con 0x7fb00c0a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:36.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb011185700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb00c097360 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:36.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb011185700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb00c0980e0 con 0x7fb00c097360 2023-09-30T01:54:36.544 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb011185700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:36.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb011185700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb00c0991a0 con 0x7fb00c096a50 2023-09-30T01:54:36.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb011185700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:36.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb011185700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c099480 con 0x7fb00c0a2390 2023-09-30T01:54:36.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb011185700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:36.545 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb011185700 10 monclient(hunting): authenticate will time out at 1166.760100s 2023-09-30T01:54:36.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb00ad9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb00c096a50 0x7fb00c1327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:36.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb00ad9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb00c096a50 0x7fb00c1327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59734/0 (socket says 172.21.15.50:59734) 2023-09-30T01:54:36.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb00ad9d700 1 -- 172.21.15.50:0/2054801424 learned_addr learned my addr 172.21.15.50:0/2054801424 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-09-30T01:54:36.546 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb00ad9d700 10 monclient(hunting): get_auth_request con 0x7fb00c096a50 auth_method 0 2023-09-30T01:54:36.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb00ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:36.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb00ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:36.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb00ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:36.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb00ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:36.547 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb00ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:36.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb00ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:36.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb00a59c700 1 --2- 172.21.15.50:0/2054801424 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb00c0a2390 0x7fb00c096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:36.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb00ad9d700 10 monclient(hunting): handle_auth_done global_id 4132 payload 274 2023-09-30T01:54:36.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb00ad9d700 1 -- 172.21.15.50:0/2054801424 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb00c0a2390 msgr2=0x7fb00c096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:36.548 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb00ad9d700 1 --2- 172.21.15.50:0/2054801424 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb00c0a2390 0x7fb00c096e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb00ad9d700 1 -- 172.21.15.50:0/2054801424 >> v1:172.21.15.7:6789/0 conn(0x7fb00c097360 legacy=0x7fb00c12ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:36.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb00ad9d700 10 monclient: _finish_hunting 0 2023-09-30T01:54:36.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb00ad9d700 1 monclient: found mon.noname-c 2023-09-30T01:54:36.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb00ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:36.549 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb00ad9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3300/0 2023-09-30T01:54:36.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.532+0000 7fb00ad9d700 1 -- 172.21.15.50:0/2054801424 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00c097bf0 con 0x7fb00c096a50 2023-09-30T01:54:36.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.534+0000 7fb00ad9d700 1 -- 172.21.15.50:0/2054801424 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb00c096a50 msgr2=0x7fb00c1327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:36.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.534+0000 7fb00ad9d700 1 --2- 172.21.15.50:0/2054801424 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb00c096a50 0x7fb00c1327a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7faffc0099b0 tx=0x7faffc009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=379a1befeed89fb5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:36.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.534+0000 7fb009d9b700 1 -- 172.21.15.50:0/2054801424 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faffc00bb10 con 0x7fb00c096a50 2023-09-30T01:54:36.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.534+0000 7fb009d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:36.550 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.534+0000 7fb009d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:36.551 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.534+0000 7fb009d9b700 10 monclient: dump: 2023-09-30T01:54:36.551 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:36.551 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:36.551 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.551 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.551 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:36.552 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:36.552 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:36.552 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:36.552 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:36.552 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:36.552 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.534+0000 7fb009d9b700 10 monclient: _finish_auth 0 2023-09-30T01:54:36.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.534+0000 7fb009d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:36.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.534+0000 7fb009d9b700 1 -- 172.21.15.50:0/2054801424 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faffc00bc70 con 0x7fb00c096a50 2023-09-30T01:54:36.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.534+0000 7fb011185700 5 monclient: authenticate success, global_id 4132 2023-09-30T01:54:36.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.534+0000 7fb011185700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:36.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.534+0000 7fb009d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:36.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.534+0000 7fb009d9b700 1 -- 172.21.15.50:0/2054801424 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faffc011420 con 0x7fb00c096a50 2023-09-30T01:54:36.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.534+0000 7fb011185700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:36.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.534+0000 7fb011185700 4 set_mon_vals no callback set 2023-09-30T01:54:36.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.534+0000 7fb009d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:36.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.534+0000 7fb009d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:54:36.553 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.534+0000 7fb009d9b700 10 monclient: dump: 2023-09-30T01:54:36.554 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:36.554 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:36.554 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.554 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.554 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:36.554 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:36.554 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:36.554 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:36.554 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:36.555 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:36.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.535+0000 7fb011185700 10 monclient: shutdown 2023-09-30T01:54:36.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.535+0000 7fb011185700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:36.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.535+0000 7fb011185700 1 -- 172.21.15.50:0/2054801424 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb00c096a50 msgr2=0x7fb00c1327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:36.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.535+0000 7fb011185700 1 --2- 172.21.15.50:0/2054801424 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb00c096a50 0x7fb00c1327a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7faffc0099b0 tx=0x7faffc009250 comp rx=0 tx=0).stop 2023-09-30T01:54:36.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.535+0000 7fb011185700 1 -- 172.21.15.50:0/2054801424 shutdown_connections 2023-09-30T01:54:36.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.535+0000 7fb011185700 1 --2- 172.21.15.50:0/2054801424 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb00c096a50 0x7fb00c1327a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.535+0000 7fb011185700 1 --2- 172.21.15.50:0/2054801424 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb00c0a2390 0x7fb00c096e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.535+0000 7fb011185700 1 -- 172.21.15.50:0/2054801424 >> 172.21.15.50:0/2054801424 conn(0x7fb00c070cf0 msgr2=0x7fb00c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:36.555 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.535+0000 7fb011185700 1 -- 172.21.15.50:0/2054801424 shutdown_connections 2023-09-30T01:54:36.556 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.535+0000 7fb011185700 1 -- 172.21.15.50:0/2054801424 wait complete. 2023-09-30T01:54:36.556 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:36.536+0000 7fc92c365180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T01:54:36.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.536+0000 7fb011185700 10 monclient: build_initial_monmap 2023-09-30T01:54:36.561 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.536+0000 7fb011185700 10 monclient: monmap: 2023-09-30T01:54:36.561 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2023-09-30T01:54:36.561 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:36.561 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2023-09-30T01:54:36.561 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2023-09-30T01:54:36.561 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:36.561 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:36.561 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:36.561 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:36.561 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:36.562 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:36.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.536+0000 7fb011185700 1 Processor -- start 2023-09-30T01:54:36.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.536+0000 7fb011185700 1 -- start start 2023-09-30T01:54:36.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.536+0000 7fb011185700 10 monclient: init 2023-09-30T01:54:36.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb011185700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:36.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb011185700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:54:36.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb011185700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb00c096a50 0x7fb00c12dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:36.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb011185700 10 monclient(hunting): picked mon.noname-c con 0x7fb00c096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:36.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb011185700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb00c097360 0x7fb00c09dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:36.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb011185700 10 monclient(hunting): picked mon.noname-b con 0x7fb00c097360 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:36.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb011185700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb00c0a2390 0x7fb00c09e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:36.562 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb011185700 10 monclient(hunting): picked mon.noname-a con 0x7fb00c0a2390 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:36.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb011185700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:36.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb011185700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c12f030 con 0x7fb00c0a2390 2023-09-30T01:54:36.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb011185700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:36.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb011185700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c0a1530 con 0x7fb00c097360 2023-09-30T01:54:36.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb011185700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:36.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb011185700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c0a1250 con 0x7fb00c096a50 2023-09-30T01:54:36.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb011185700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:36.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb011185700 10 monclient(hunting): authenticate will time out at 1166.765130s 2023-09-30T01:54:36.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb00c097360 0x7fb00c09dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:36.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb00c097360 0x7fb00c09dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59740/0 (socket says 172.21.15.50:59740) 2023-09-30T01:54:36.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00a59c700 1 -- 172.21.15.50:0/3257995631 learned_addr learned my addr 172.21.15.50:0/3257995631 (peer_addr_for_me v2:172.21.15.50:0/0) 2023-09-30T01:54:36.563 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00a59c700 10 monclient(hunting): get_auth_request con 0x7fb00c097360 auth_method 0 2023-09-30T01:54:36.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:36.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:36.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:36.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00ad9d700 1 --2- 172.21.15.50:0/3257995631 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb00c096a50 0x7fb00c12dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:36.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00b59e700 1 --2- 172.21.15.50:0/3257995631 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb00c0a2390 0x7fb00c09e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:36.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:36.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:36.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:36.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00ad9d700 10 monclient(hunting): get_auth_request con 0x7fb00c096a50 auth_method 0 2023-09-30T01:54:36.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:36.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:36.564 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:36.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00b59e700 10 monclient(hunting): get_auth_request con 0x7fb00c0a2390 auth_method 0 2023-09-30T01:54:36.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00b59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:36.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00b59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:36.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00b59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:36.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00b59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:36.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00b59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:36.565 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00b59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:36.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:36.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:36.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:36.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00a59c700 10 monclient(hunting): handle_auth_done global_id 4135 payload 824 2023-09-30T01:54:36.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00a59c700 1 -- 172.21.15.50:0/3257995631 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb00c096a50 msgr2=0x7fb00c12dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:36.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00a59c700 1 --2- 172.21.15.50:0/3257995631 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb00c096a50 0x7fb00c12dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00a59c700 1 -- 172.21.15.50:0/3257995631 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb00c0a2390 msgr2=0x7fb00c09e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:36.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00a59c700 1 --2- 172.21.15.50:0/3257995631 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb00c0a2390 0x7fb00c09e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00a59c700 10 monclient: _finish_hunting 0 2023-09-30T01:54:36.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00a59c700 1 monclient: found mon.noname-b 2023-09-30T01:54:36.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:36.566 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00a59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3300/0 2023-09-30T01:54:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.537+0000 7fb00a59c700 1 -- 172.21.15.50:0/3257995631 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc0094f0 con 0x7fb00c097360 2023-09-30T01:54:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.539+0000 7fb00a59c700 1 -- 172.21.15.50:0/3257995631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb00c097360 msgr2=0x7fb00c09dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.539+0000 7fb00a59c700 1 --2- 172.21.15.50:0/3257995631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb00c097360 0x7fb00c09dc20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb00000c880 tx=0x7fb000008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7faff2ffd700 1 -- 172.21.15.50:0/3257995631 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb000011810 con 0x7fb00c097360 2023-09-30T01:54:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7faff2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7faff2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:54:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7faff2ffd700 10 monclient: dump: 2023-09-30T01:54:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:36.569 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.569 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.569 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:36.569 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:36.569 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:36.569 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:36.569 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:36.570 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7faff2ffd700 10 monclient: _finish_auth 0 2023-09-30T01:54:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7faff2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7fb011185700 5 monclient: authenticate success, global_id 4135 2023-09-30T01:54:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7faff2ffd700 1 -- 172.21.15.50:0/3257995631 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb000011970 con 0x7fb00c097360 2023-09-30T01:54:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7faff2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7fb011185700 10 monclient: _renew_subs 2023-09-30T01:54:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7faff2ffd700 1 -- 172.21.15.50:0/3257995631 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb000011d30 con 0x7fb00c097360 2023-09-30T01:54:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7fb011185700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7fb011185700 1 -- 172.21.15.50:0/3257995631 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb00c09f7b0 con 0x7fb00c097360 2023-09-30T01:54:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7faff2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7faff2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:54:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7faff2ffd700 10 monclient: dump: 2023-09-30T01:54:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2023-09-30T01:54:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2023-09-30T01:54:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:36.572 INFO:teuthology.orchestra.run.smithi050.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:36.572 INFO:teuthology.orchestra.run.smithi050.stderr: 2023-09-30T01:54:36.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7fb00959a700 4 set_mon_vals no callback set 2023-09-30T01:54:36.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7fb011185700 10 monclient: _renew_subs 2023-09-30T01:54:36.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7fb011185700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:36.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7fb011185700 1 -- 172.21.15.50:0/3257995631 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb00c136ec0 con 0x7fb00c097360 2023-09-30T01:54:36.572 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7faff2ffd700 1 -- 172.21.15.50:0/3257995631 <== mon.1 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fb000019840 con 0x7fb00c097360 2023-09-30T01:54:36.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.540+0000 7faff2ffd700 1 -- 172.21.15.50:0/3257995631 <== mon.1 v2:172.21.15.50:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7fb00001aab0 con 0x7fb00c097360 2023-09-30T01:54:36.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.541+0000 7fb011185700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:36.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.541+0000 7fb011185700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:36.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.541+0000 7fb011185700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:36.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.541+0000 7fb011185700 1 -- 172.21.15.50:0/3257995631 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb00c0a48f0 con 0x7fb00c097360 2023-09-30T01:54:36.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.543+0000 7faff2ffd700 1 -- 172.21.15.50:0/3257995631 <== mon.1 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fb0000396d0 con 0x7fb00c097360 2023-09-30T01:54:36.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.543+0000 7faff2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:36.573 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.543+0000 7faff2ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:36.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.626+0000 7fb011185700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e6a847ab-276e-4d54-9a8f-6df5b8ab54cc", "id": 5}] 2023-09-30T01:54:36.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.627+0000 7fb011185700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e6a847ab-276e-4d54-9a8f-6df5b8ab54cc", "id": 5}] 2023-09-30T01:54:36.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.627+0000 7fb011185700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:36.627 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.627+0000 7fb011185700 1 -- 172.21.15.50:0/3257995631 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e6a847ab-276e-4d54-9a8f-6df5b8ab54cc", "id": 5} v 0) v1 -- 0x7fb00c064c10 con 0x7fb00c097360 2023-09-30T01:54:36.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.630+0000 7faff2ffd700 1 -- 172.21.15.50:0/3257995631 <== mon.1 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e6a847ab-276e-4d54-9a8f-6df5b8ab54cc", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb000028070 con 0x7fb00c097360 2023-09-30T01:54:36.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.630+0000 7faff2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e6a847ab-276e-4d54-9a8f-6df5b8ab54cc", "id": 5}] 2023-09-30T01:54:36.630 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.630+0000 7faff2ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:36.630 INFO:teuthology.orchestra.run.smithi050.stdout:5 2023-09-30T01:54:36.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.631+0000 7fb011185700 10 monclient: shutdown 2023-09-30T01:54:36.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.631+0000 7fb011185700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:36.631 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.631+0000 7fb011185700 1 -- 172.21.15.50:0/3257995631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb00c097360 msgr2=0x7fb00c09dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:36.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.631+0000 7fb011185700 1 --2- 172.21.15.50:0/3257995631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb00c097360 0x7fb00c09dc20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb00000c880 tx=0x7fb000008040 comp rx=0 tx=0).stop 2023-09-30T01:54:36.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.631+0000 7fb011185700 1 -- 172.21.15.50:0/3257995631 shutdown_connections 2023-09-30T01:54:36.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.631+0000 7fb011185700 1 --2- 172.21.15.50:0/3257995631 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb00c0a2390 0x7fb00c09e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.631+0000 7fb011185700 1 --2- 172.21.15.50:0/3257995631 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb00c096a50 0x7fb00c12dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.631+0000 7fb011185700 1 --2- 172.21.15.50:0/3257995631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb00c097360 0x7fb00c09dc20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.631+0000 7fb011185700 1 -- 172.21.15.50:0/3257995631 >> 172.21.15.50:0/3257995631 conn(0x7fb00c070cf0 msgr2=0x7fb00c1310a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:36.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.631+0000 7fb011185700 1 -- 172.21.15.50:0/3257995631 shutdown_connections 2023-09-30T01:54:36.632 INFO:teuthology.orchestra.run.smithi050.stderr:2023-09-30T01:54:36.631+0000 7fb011185700 1 -- 172.21.15.50:0/3257995631 wait complete. 2023-09-30T01:54:36.637 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:36.637+0000 7fc92c365180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T01:54:36.642 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-30T01:54:36.643 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-09-30T01:54:36.645 INFO:tasks.ceph.osd.0:Started 2023-09-30T01:54:36.645 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-30T01:54:36.646 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-09-30T01:54:36.648 INFO:tasks.ceph.osd.1:Started 2023-09-30T01:54:36.648 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-30T01:54:36.648 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-09-30T01:54:36.650 INFO:tasks.ceph.osd.2:Started 2023-09-30T01:54:36.650 INFO:tasks.ceph.osd.3:Restarting daemon 2023-09-30T01:54:36.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-09-30T01:54:36.653 INFO:tasks.ceph.osd.3:Started 2023-09-30T01:54:36.653 INFO:tasks.ceph.osd.4:Restarting daemon 2023-09-30T01:54:36.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-09-30T01:54:36.655 INFO:tasks.ceph.osd.4:Started 2023-09-30T01:54:36.655 INFO:tasks.ceph.osd.5:Restarting daemon 2023-09-30T01:54:36.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-09-30T01:54:36.657 INFO:tasks.ceph.osd.5:Started 2023-09-30T01:54:36.658 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:54:36.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.800+0000 7f45f25d6700 10 monclient: get_monmap_and_config 2023-09-30T01:54:36.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.800+0000 7f45f25d6700 10 monclient: build_initial_monmap 2023-09-30T01:54:36.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.800+0000 7f45f25d6700 10 monclient: monmap: 2023-09-30T01:54:36.897 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:36.897 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:36.897 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:36.801216+0000 2023-09-30T01:54:36.897 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:36.801216+0000 2023-09-30T01:54:36.897 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:36.897 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:36.897 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:36.898 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:36.898 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:36.898 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:36.898 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:36.898 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:36.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.800+0000 7f45f25d6700 1 Processor -- start 2023-09-30T01:54:36.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.800+0000 7f45f25d6700 1 -- start start 2023-09-30T01:54:36.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.800+0000 7f45f25d6700 10 monclient: init 2023-09-30T01:54:36.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45f25d6700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:36.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45f25d6700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-09-30T01:54:36.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45f25d6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f45ec096a50 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:36.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45f25d6700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45ec096e20 0x7f45ec0971f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:36.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45f25d6700 10 monclient(hunting): picked mon.noname-b con 0x7f45ec096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:36.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45f25d6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f45ec137710 addr v1:172.21.15.50:6789/0 2023-09-30T01:54:36.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45f25d6700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45ec132d90 con 0x7f45ec096a50 2023-09-30T01:54:36.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45f25d6700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45ec097b40 con 0x7f45ec137710 2023-09-30T01:54:36.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45f25d6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:36.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45f25d6700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec099160 con 0x7f45ec096e20 2023-09-30T01:54:36.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45f25d6700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:36.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45f25d6700 10 monclient(hunting): authenticate will time out at 1165.866412s 2023-09-30T01:54:36.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45ebfff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f45ec096a50 0x7f45ec132680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42736/0 (socket says 172.21.15.7:42736) 2023-09-30T01:54:36.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45ebfff700 1 -- 172.21.15.7:0/1615044712 learned_addr learned my addr 172.21.15.7:0/1615044712 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:54:36.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45eb7fe700 1 --2- 172.21.15.7:0/1615044712 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45ec096e20 0x7f45ec0971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:36.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45eb7fe700 10 monclient(hunting): get_auth_request con 0x7f45ec096e20 auth_method 0 2023-09-30T01:54:36.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45eb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:36.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45eb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:36.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45eb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:36.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45eb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:36.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45eb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:36.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45eb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:36.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45eb7fe700 10 monclient(hunting): handle_auth_done global_id 4139 payload 274 2023-09-30T01:54:36.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45eb7fe700 1 -- 172.21.15.7:0/1615044712 >> v1:172.21.15.50:6789/0 conn(0x7f45ec137710 legacy=0x7f45ec139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:36.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45eb7fe700 1 -- 172.21.15.7:0/1615044712 >> v1:172.21.15.7:6789/0 conn(0x7f45ec096a50 legacy=0x7f45ec132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:36.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45eb7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:54:36.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45eb7fe700 1 monclient: found mon.noname-b 2023-09-30T01:54:36.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45eb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:36.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45eb7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:54:36.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.801+0000 7f45eb7fe700 1 -- 172.21.15.7:0/1615044712 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ec099720 con 0x7f45ec096e20 2023-09-30T01:54:36.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45eb7fe700 1 -- 172.21.15.7:0/1615044712 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45ec096e20 msgr2=0x7f45ec0971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:36.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45eb7fe700 1 --2- 172.21.15.7:0/1615044712 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45ec096e20 0x7f45ec0971f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f45e0009910 tx=0x7f45e00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed8ddd7dc35d0495 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:36.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45eaffd700 1 -- 172.21.15.7:0/1615044712 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e000b9f0 con 0x7f45ec096e20 2023-09-30T01:54:36.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45eaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:36.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45eaffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:54:36.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45eaffd700 10 monclient: dump: 2023-09-30T01:54:36.910 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:36.910 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:36.910 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.910 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.910 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:36.910 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:36.910 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:36.910 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:36.911 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:36.911 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:36.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45eaffd700 10 monclient: _finish_auth 0 2023-09-30T01:54:36.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45eaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:36.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45eaffd700 1 -- 172.21.15.7:0/1615044712 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e000bb50 con 0x7f45ec096e20 2023-09-30T01:54:36.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45f25d6700 5 monclient: authenticate success, global_id 4139 2023-09-30T01:54:36.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45f25d6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:36.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45eaffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:36.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45eaffd700 1 -- 172.21.15.7:0/1615044712 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e0011420 con 0x7f45ec096e20 2023-09-30T01:54:36.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45f25d6700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:36.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45f25d6700 4 set_mon_vals no callback set 2023-09-30T01:54:36.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45eaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:36.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45eaffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:36.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45eaffd700 10 monclient: dump: 2023-09-30T01:54:36.912 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:36.912 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:36.912 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.912 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.912 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:36.912 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:36.912 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:36.912 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:36.912 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:36.913 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:36.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45f25d6700 10 monclient: shutdown 2023-09-30T01:54:36.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45f25d6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:36.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45f25d6700 1 -- 172.21.15.7:0/1615044712 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45ec096e20 msgr2=0x7f45ec0971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:36.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45f25d6700 1 --2- 172.21.15.7:0/1615044712 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45ec096e20 0x7f45ec0971f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f45e0009910 tx=0x7f45e00091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45f25d6700 1 -- 172.21.15.7:0/1615044712 shutdown_connections 2023-09-30T01:54:36.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45f25d6700 1 --2- 172.21.15.7:0/1615044712 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45ec096e20 0x7f45ec0971f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45f25d6700 1 -- 172.21.15.7:0/1615044712 >> 172.21.15.7:0/1615044712 conn(0x7f45ec070cf0 msgr2=0x7f45ec0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:36.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45f25d6700 1 -- 172.21.15.7:0/1615044712 shutdown_connections 2023-09-30T01:54:36.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.804+0000 7f45f25d6700 1 -- 172.21.15.7:0/1615044712 wait complete. 2023-09-30T01:54:36.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.805+0000 7f45f25d6700 10 monclient: build_initial_monmap 2023-09-30T01:54:36.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.805+0000 7f45f25d6700 10 monclient: monmap: 2023-09-30T01:54:36.914 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:36.914 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:36.914 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:36.914 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:36.914 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:36.914 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:36.914 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:36.914 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:36.914 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:36.914 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:36.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.805+0000 7f45f25d6700 1 Processor -- start 2023-09-30T01:54:36.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.805+0000 7f45f25d6700 1 -- start start 2023-09-30T01:54:36.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.805+0000 7f45f25d6700 10 monclient: init 2023-09-30T01:54:36.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f25d6700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:36.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f25d6700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:54:36.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f25d6700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45ec096a50 0x7f45ec0a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:36.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f25d6700 10 monclient(hunting): picked mon.noname-c con 0x7f45ec096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:36.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f25d6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45ec096e20 0x7f45ec09cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:36.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f25d6700 10 monclient(hunting): picked mon.noname-b con 0x7f45ec096e20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:36.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f25d6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45ec137710 0x7f45ec09d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:36.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f25d6700 10 monclient(hunting): picked mon.noname-a con 0x7f45ec137710 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:36.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f25d6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:36.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f25d6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec0a08d0 con 0x7f45ec137710 2023-09-30T01:54:36.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f25d6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:36.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f25d6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec1334e0 con 0x7f45ec096e20 2023-09-30T01:54:36.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f25d6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:36.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f25d6700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45ec131d20 con 0x7f45ec096a50 2023-09-30T01:54:36.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f25d6700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:36.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f25d6700 10 monclient(hunting): authenticate will time out at 1165.871467s 2023-09-30T01:54:36.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45ebfff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45ec096a50 0x7f45ec0a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:36.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45ebfff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45ec096a50 0x7f45ec0a1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33478/0 (socket says 172.21.15.7:33478) 2023-09-30T01:54:36.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45ebfff700 1 -- 172.21.15.7:0/1462237494 learned_addr learned my addr 172.21.15.7:0/1462237494 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:36.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45ebfff700 10 monclient(hunting): get_auth_request con 0x7f45ec096a50 auth_method 0 2023-09-30T01:54:36.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45ebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:36.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45ebfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:36.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45ebfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:36.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f0b73700 1 --2- 172.21.15.7:0/1462237494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45ec137710 0x7f45ec09d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:36.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f0b73700 10 monclient(hunting): get_auth_request con 0x7f45ec137710 auth_method 0 2023-09-30T01:54:36.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f0b73700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:36.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f0b73700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:36.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f0b73700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:36.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45eb7fe700 1 --2- 172.21.15.7:0/1462237494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45ec096e20 0x7f45ec09cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:36.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45ebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:36.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45ebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:36.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45ebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:36.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45eb7fe700 10 monclient(hunting): get_auth_request con 0x7f45ec096e20 auth_method 0 2023-09-30T01:54:36.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45eb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45eb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45eb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f0b73700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f0b73700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f0b73700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45eb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45eb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45eb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45ebfff700 10 monclient(hunting): handle_auth_done global_id 4142 payload 824 2023-09-30T01:54:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45ebfff700 1 -- 172.21.15.7:0/1462237494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45ec096e20 msgr2=0x7f45ec09cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45ebfff700 1 --2- 172.21.15.7:0/1462237494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45ec096e20 0x7f45ec09cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45ebfff700 1 -- 172.21.15.7:0/1462237494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45ec137710 msgr2=0x7f45ec09d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45ebfff700 1 --2- 172.21.15.7:0/1462237494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45ec137710 0x7f45ec09d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45ebfff700 10 monclient: _finish_hunting 0 2023-09-30T01:54:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45ebfff700 1 monclient: found mon.noname-c 2023-09-30T01:54:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45ebfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45ebfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:54:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45ebfff700 1 -- 172.21.15.7:0/1462237494 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e0009420 con 0x7f45ec096a50 2023-09-30T01:54:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.806+0000 7f45f0b73700 1 --2- 172.21.15.7:0/1462237494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45ec137710 0x7f45ec09d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45ebfff700 1 -- 172.21.15.7:0/1462237494 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45ec096a50 msgr2=0x7f45ec0a1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45ebfff700 1 --2- 172.21.15.7:0/1462237494 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45ec096a50 0x7f45ec0a1290 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f45dc00c640 tx=0x7f45dc004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45d3fff700 1 -- 172.21.15.7:0/1462237494 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45dc005d40 con 0x7f45ec096a50 2023-09-30T01:54:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45d3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:36.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45d3fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:36.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45d3fff700 10 monclient: dump: 2023-09-30T01:54:36.920 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:36.920 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:36.920 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.920 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.920 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:36.920 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:36.920 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:36.920 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:36.920 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:36.920 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:36.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45d3fff700 10 monclient: _finish_auth 0 2023-09-30T01:54:36.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45d3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:36.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45f25d6700 5 monclient: authenticate success, global_id 4142 2023-09-30T01:54:36.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45d3fff700 1 -- 172.21.15.7:0/1462237494 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45dc0070d0 con 0x7f45ec096a50 2023-09-30T01:54:36.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45f25d6700 10 monclient: _renew_subs 2023-09-30T01:54:36.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45f25d6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:36.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45f25d6700 1 -- 172.21.15.7:0/1462237494 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45ec09ecd0 con 0x7f45ec096a50 2023-09-30T01:54:36.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45d3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:36.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45d3fff700 1 -- 172.21.15.7:0/1462237494 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45dc007490 con 0x7f45ec096a50 2023-09-30T01:54:36.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45d3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:36.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45d3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:36.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45d3fff700 10 monclient: dump: 2023-09-30T01:54:36.921 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:36.921 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:36.922 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.922 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:36.922 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:36.922 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:36.922 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:36.922 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:36.922 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:36.922 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:36.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45ea7fc700 4 set_mon_vals no callback set 2023-09-30T01:54:36.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45f25d6700 10 monclient: _renew_subs 2023-09-30T01:54:36.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45f25d6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:36.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45f25d6700 1 -- 172.21.15.7:0/1462237494 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45ec09e750 con 0x7f45ec096a50 2023-09-30T01:54:36.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45d3fff700 1 -- 172.21.15.7:0/1462237494 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f45dc01c6c0 con 0x7f45ec096a50 2023-09-30T01:54:36.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.809+0000 7f45d3fff700 1 -- 172.21.15.7:0/1462237494 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f45dc012a00 con 0x7f45ec096a50 2023-09-30T01:54:36.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.810+0000 7f45f25d6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:36.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.810+0000 7f45f25d6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:36.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.810+0000 7f45f25d6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:36.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.810+0000 7f45f25d6700 1 -- 172.21.15.7:0/1462237494 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45ec064c10 con 0x7f45ec096a50 2023-09-30T01:54:36.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.812+0000 7f45d3fff700 1 -- 172.21.15.7:0/1462237494 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f45dc017070 con 0x7f45ec096a50 2023-09-30T01:54:36.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.812+0000 7f45d3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:36.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.812+0000 7f45d3fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:36.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.881+0000 7f45f25d6700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:54:36.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.881+0000 7f45f25d6700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:54:36.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.881+0000 7f45f25d6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:36.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.881+0000 7f45f25d6700 1 -- 172.21.15.7:0/1462237494 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45ec097730 con 0x7f45ec096a50 2023-09-30T01:54:36.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.881+0000 7f45d3fff700 1 -- 172.21.15.7:0/1462237494 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5560 (secure 0 0 0) 0x7f45dc01cba0 con 0x7f45ec096a50 2023-09-30T01:54:36.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.881+0000 7f45d3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:54:36.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.881+0000 7f45d3fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:36.924 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:36.924 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":8,"fsid":"2ea1044d-d984-49b9-baad-7f36bd454a19","created":"2023-09-30T01:54:34.356209+0000","modified":"2023-09-30T01:54:36.628886+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T01:54:36.628886+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"e44e9247-cb86-4a16-ac1f-f021cdf5108f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e30730fd-1e49-44bf-bdc9-087e23727df0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b39c552e-f5cc-4e4f-90a3-5aa5f5127a27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"15cd89ae-049a-4e1a-8c6b-7d086ca6e1df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"de6773bd-d286-4ab9-b91f-8a90ee3073d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e6a847ab-276e-4d54-9a8f-6df5b8ab54cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:54:36.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.882+0000 7f45f25d6700 10 monclient: shutdown 2023-09-30T01:54:36.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.882+0000 7f45f25d6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:36.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.882+0000 7f45f25d6700 1 -- 172.21.15.7:0/1462237494 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45ec096a50 msgr2=0x7f45ec0a1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:36.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.882+0000 7f45f25d6700 1 --2- 172.21.15.7:0/1462237494 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45ec096a50 0x7f45ec0a1290 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f45dc00c640 tx=0x7f45dc004a80 comp rx=0 tx=0).stop 2023-09-30T01:54:36.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.882+0000 7f45f25d6700 1 -- 172.21.15.7:0/1462237494 shutdown_connections 2023-09-30T01:54:36.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.882+0000 7f45f25d6700 1 --2- 172.21.15.7:0/1462237494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f45ec137710 0x7f45ec09d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.882+0000 7f45f25d6700 1 --2- 172.21.15.7:0/1462237494 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f45ec096a50 0x7f45ec0a1290 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.882+0000 7f45f25d6700 1 --2- 172.21.15.7:0/1462237494 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f45ec096e20 0x7f45ec09cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:36.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.882+0000 7f45f25d6700 1 -- 172.21.15.7:0/1462237494 >> 172.21.15.7:0/1462237494 conn(0x7f45ec070cf0 msgr2=0x7f45ec12f110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:36.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.882+0000 7f45f25d6700 1 -- 172.21.15.7:0/1462237494 shutdown_connections 2023-09-30T01:54:36.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:36.882+0000 7f45f25d6700 1 -- 172.21.15.7:0/1462237494 wait complete. 2023-09-30T01:54:36.927 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-30T01:54:36.927 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-30T01:54:37.185 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:37.184+0000 7fc92c365180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T01:54:37.258 INFO:tasks.ceph.osd.5.smithi050.stderr:2023-09-30T01:54:37.257+0000 7fa51d706540 -1 Falling back to public interface 2023-09-30T01:54:37.258 INFO:tasks.ceph.osd.3.smithi050.stderr:2023-09-30T01:54:37.258+0000 7f7f853b0540 -1 Falling back to public interface 2023-09-30T01:54:37.259 INFO:tasks.ceph.osd.4.smithi050.stderr:2023-09-30T01:54:37.258+0000 7f24d6943540 -1 Falling back to public interface 2023-09-30T01:54:37.260 INFO:tasks.ceph.osd.3.smithi050.stderr:2023-09-30T01:54:37.258+0000 7f7f7f7d8700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6802/0: (98) Address already in use 2023-09-30T01:54:37.260 INFO:tasks.ceph.osd.3.smithi050.stderr:2023-09-30T01:54:37.258+0000 7f7f7f7d8700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6803/0: (98) Address already in use 2023-09-30T01:54:37.260 INFO:tasks.ceph.osd.4.smithi050.stderr:2023-09-30T01:54:37.259+0000 7f24d0d6b700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6813/0: (98) Address already in use 2023-09-30T01:54:37.261 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-09-30T01:54:37.257+0000 7f179de1b540 -1 Falling back to public interface 2023-09-30T01:54:37.262 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-09-30T01:54:37.262+0000 7fcc43007540 -1 Falling back to public interface 2023-09-30T01:54:37.266 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-09-30T01:54:37.266+0000 7f35e7ff6540 -1 Falling back to public interface 2023-09-30T01:54:37.267 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:37.268+0000 7fc92c365180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T01:54:37.349 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:37.349+0000 7fc92c365180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T01:54:37.533 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T01:54:37.648 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.648+0000 7fd4e0022700 10 monclient: get_monmap_and_config 2023-09-30T01:54:37.648 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.648+0000 7fd4e0022700 10 monclient: build_initial_monmap 2023-09-30T01:54:37.649 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.648+0000 7fd4e0022700 10 monclient: monmap: 2023-09-30T01:54:37.649 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2023-09-30T01:54:37.649 INFO:teuthology.misc.health.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:37.649 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-09-30T01:54:37.649602+0000 2023-09-30T01:54:37.649 INFO:teuthology.misc.health.smithi007.stderr:created 2023-09-30T01:54:37.649602+0000 2023-09-30T01:54:37.649 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:37.649 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:37.649 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:37.649 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:37.649 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:37.650 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:37.650 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:37.650 INFO:teuthology.misc.health.smithi007.stderr: 2023-09-30T01:54:37.654 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.649+0000 7fd4e0022700 1 Processor -- start 2023-09-30T01:54:37.654 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.649+0000 7fd4e0022700 1 -- start start 2023-09-30T01:54:37.654 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.649+0000 7fd4e0022700 10 monclient: init 2023-09-30T01:54:37.654 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.649+0000 7fd4e0022700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:37.654 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.649+0000 7fd4e0022700 10 monclient: _add_conns ranks=[3,2,0,1,4] 2023-09-30T01:54:37.654 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.649+0000 7fd4e0022700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd4d809ffe0 0x7fd4d809ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:37.654 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.649+0000 7fd4e0022700 10 monclient(hunting): picked mon.noname-a con 0x7fd4d809ffe0 addr v2:172.21.15.7:3300/0 2023-09-30T01:54:37.654 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.649+0000 7fd4e0022700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd4d809efd0 0x7fd4d80a03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:37.654 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.649+0000 7fd4e0022700 10 monclient(hunting): picked mon.noname-b con 0x7fd4d809efd0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:37.655 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.649+0000 7fd4e0022700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd4d80a08f0 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:37.655 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.649+0000 7fd4e0022700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d8064fd0 con 0x7fd4d80a08f0 2023-09-30T01:54:37.655 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.649+0000 7fd4e0022700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:37.655 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.649+0000 7fd4e0022700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4d80a1de0 con 0x7fd4d809ffe0 2023-09-30T01:54:37.655 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.649+0000 7fd4e0022700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:37.655 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.649+0000 7fd4e0022700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d809e040 con 0x7fd4d809efd0 2023-09-30T01:54:37.655 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.649+0000 7fd4e0022700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:37.655 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.649+0000 7fd4e0022700 10 monclient(hunting): authenticate will time out at 1166.714837s 2023-09-30T01:54:37.655 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.649+0000 7fd4dd5bd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd4d809efd0 0x7fd4d80a03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:37.655 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.649+0000 7fd4dddbe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd4d809ffe0 0x7fd4d809ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:37.656 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.649+0000 7fd4de5bf700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd4d80a08f0 0x7fd4d80984b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42752/0 (socket says 172.21.15.7:42752) 2023-09-30T01:54:37.656 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4de5bf700 1 -- 172.21.15.7:0/4278068077 learned_addr learned my addr 172.21.15.7:0/4278068077 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:54:37.656 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dddbe700 10 monclient(hunting): get_auth_request con 0x7fd4d809ffe0 auth_method 0 2023-09-30T01:54:37.656 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dddbe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:37.656 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dddbe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:37.656 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dddbe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:37.656 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dd5bd700 10 monclient(hunting): get_auth_request con 0x7fd4d809efd0 auth_method 0 2023-09-30T01:54:37.656 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dd5bd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:37.656 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dd5bd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:37.656 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dd5bd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:37.657 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dddbe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:37.657 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dddbe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:37.657 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dddbe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:37.657 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dd5bd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:37.657 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dd5bd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:37.657 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dd5bd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:37.657 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dd5bd700 10 monclient(hunting): handle_auth_done global_id 4145 payload 274 2023-09-30T01:54:37.657 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dd5bd700 1 -- 172.21.15.7:0/4278068077 >> v2:172.21.15.7:3300/0 conn(0x7fd4d809ffe0 msgr2=0x7fd4d809ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:37.657 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dd5bd700 1 --2- 172.21.15.7:0/4278068077 >> v2:172.21.15.7:3300/0 conn(0x7fd4d809ffe0 0x7fd4d809ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:37.658 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dd5bd700 1 -- 172.21.15.7:0/4278068077 >> v1:172.21.15.7:6789/0 conn(0x7fd4d80a08f0 legacy=0x7fd4d80984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:37.658 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dd5bd700 10 monclient: _finish_hunting 0 2023-09-30T01:54:37.658 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dd5bd700 1 monclient: found mon.noname-b 2023-09-30T01:54:37.658 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dd5bd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:37.658 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dd5bd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:54:37.658 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dd5bd700 1 -- 172.21.15.7:0/4278068077 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d809da80 con 0x7fd4d809efd0 2023-09-30T01:54:37.658 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.650+0000 7fd4dddbe700 1 --2- 172.21.15.7:0/4278068077 >> v2:172.21.15.7:3300/0 conn(0x7fd4d809ffe0 0x7fd4d809ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:37.658 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.652+0000 7fd4dd5bd700 1 -- 172.21.15.7:0/4278068077 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd4d809efd0 msgr2=0x7fd4d80a03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:37.658 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.652+0000 7fd4dd5bd700 1 --2- 172.21.15.7:0/4278068077 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd4d809efd0 0x7fd4d80a03b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd4c80098e0 tx=0x7fd4c80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc7251c578e788ac server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:37.659 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.652+0000 7fd4dcdbc700 1 -- 172.21.15.7:0/4278068077 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c8011840 con 0x7fd4d809efd0 2023-09-30T01:54:37.659 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.652+0000 7fd4dcdbc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:37.659 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.652+0000 7fd4dcdbc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:54:37.659 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.652+0000 7fd4dcdbc700 10 monclient: dump: 2023-09-30T01:54:37.659 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-09-30T01:54:37.659 INFO:teuthology.misc.health.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:37.659 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:37.659 INFO:teuthology.misc.health.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:37.659 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:37.659 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:37.660 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:37.660 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:37.660 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:37.660 INFO:teuthology.misc.health.smithi007.stderr: 2023-09-30T01:54:37.660 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.652+0000 7fd4dcdbc700 10 monclient: _finish_auth 0 2023-09-30T01:54:37.660 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.652+0000 7fd4dcdbc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:37.660 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.652+0000 7fd4e0022700 5 monclient: authenticate success, global_id 4145 2023-09-30T01:54:37.660 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.652+0000 7fd4dcdbc700 1 -- 172.21.15.7:0/4278068077 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4c80119a0 con 0x7fd4d809efd0 2023-09-30T01:54:37.660 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.652+0000 7fd4e0022700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:37.661 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.652+0000 7fd4dcdbc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:37.661 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.652+0000 7fd4e0022700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:37.661 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.652+0000 7fd4dcdbc700 1 -- 172.21.15.7:0/4278068077 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c8011d40 con 0x7fd4d809efd0 2023-09-30T01:54:37.661 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.652+0000 7fd4e0022700 4 set_mon_vals no callback set 2023-09-30T01:54:37.661 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.652+0000 7fd4dcdbc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:37.661 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.652+0000 7fd4dcdbc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:37.661 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.652+0000 7fd4dcdbc700 10 monclient: dump: 2023-09-30T01:54:37.661 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-09-30T01:54:37.661 INFO:teuthology.misc.health.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:37.662 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:37.662 INFO:teuthology.misc.health.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:37.662 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:37.662 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:37.662 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:37.662 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:37.662 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:37.662 INFO:teuthology.misc.health.smithi007.stderr: 2023-09-30T01:54:37.662 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.653+0000 7fd4e0022700 10 monclient: shutdown 2023-09-30T01:54:37.662 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.653+0000 7fd4e0022700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:37.663 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.653+0000 7fd4e0022700 1 -- 172.21.15.7:0/4278068077 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd4d809efd0 msgr2=0x7fd4d80a03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:37.663 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.653+0000 7fd4e0022700 1 --2- 172.21.15.7:0/4278068077 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd4d809efd0 0x7fd4d80a03b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd4c80098e0 tx=0x7fd4c80091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:37.663 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.653+0000 7fd4e0022700 1 -- 172.21.15.7:0/4278068077 shutdown_connections 2023-09-30T01:54:37.663 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.653+0000 7fd4e0022700 1 --2- 172.21.15.7:0/4278068077 >> v2:172.21.15.7:3300/0 conn(0x7fd4d809ffe0 0x7fd4d809ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:37.663 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.653+0000 7fd4e0022700 1 --2- 172.21.15.7:0/4278068077 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd4d809efd0 0x7fd4d80a03b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:37.663 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.653+0000 7fd4e0022700 1 -- 172.21.15.7:0/4278068077 >> 172.21.15.7:0/4278068077 conn(0x7fd4d80a55f0 msgr2=0x7fd4d80a59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:37.663 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.653+0000 7fd4e0022700 1 -- 172.21.15.7:0/4278068077 shutdown_connections 2023-09-30T01:54:37.663 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.653+0000 7fd4e0022700 1 -- 172.21.15.7:0/4278068077 wait complete. 2023-09-30T01:54:37.663 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.653+0000 7fd4e0022700 10 monclient: build_initial_monmap 2023-09-30T01:54:37.664 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.653+0000 7fd4e0022700 10 monclient: monmap: 2023-09-30T01:54:37.664 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2023-09-30T01:54:37.664 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:37.664 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:37.664 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2023-09-30T01:54:37.664 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:37.664 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:37.664 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:37.664 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:37.664 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:37.665 INFO:teuthology.misc.health.smithi007.stderr: 2023-09-30T01:54:37.665 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4e0022700 1 Processor -- start 2023-09-30T01:54:37.665 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4e0022700 1 -- start start 2023-09-30T01:54:37.665 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4e0022700 10 monclient: init 2023-09-30T01:54:37.668 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4e0022700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:37.668 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4e0022700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:54:37.668 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4e0022700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4d809efd0 0x7fd4d809b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:37.669 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4e0022700 10 monclient(hunting): picked mon.noname-b con 0x7fd4d809efd0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:37.669 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4e0022700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd4d809ffe0 0x7fd4d809bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:37.669 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4e0022700 10 monclient(hunting): picked mon.noname-c con 0x7fd4d809ffe0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:37.669 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4e0022700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4d80a08f0 0x7fd4d812c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:37.669 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4e0022700 10 monclient(hunting): picked mon.noname-a con 0x7fd4d80a08f0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:37.669 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4e0022700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:37.669 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4e0022700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d806aff0 con 0x7fd4d80a08f0 2023-09-30T01:54:37.669 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4e0022700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:37.669 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4e0022700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d812fee0 con 0x7fd4d809efd0 2023-09-30T01:54:37.670 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4e0022700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:37.670 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4e0022700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d812fc00 con 0x7fd4d809ffe0 2023-09-30T01:54:37.670 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4e0022700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:37.670 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4e0022700 10 monclient(hunting): authenticate will time out at 1166.719717s 2023-09-30T01:54:37.670 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4de5bf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4d80a08f0 0x7fd4d812c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:37.670 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4dd5bd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd4d809ffe0 0x7fd4d809bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:37.670 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4dd5bd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd4d809ffe0 0x7fd4d809bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33504/0 (socket says 172.21.15.7:33504) 2023-09-30T01:54:37.670 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4de5bf700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4d80a08f0 0x7fd4d812c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53038/0 (socket says 172.21.15.7:53038) 2023-09-30T01:54:37.671 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4dd5bd700 1 -- 172.21.15.7:0/2920129776 learned_addr learned my addr 172.21.15.7:0/2920129776 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:37.671 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4dd5bd700 10 monclient(hunting): get_auth_request con 0x7fd4d809ffe0 auth_method 0 2023-09-30T01:54:37.671 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4dd5bd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:37.671 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4dd5bd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:37.671 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4dd5bd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:37.671 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4de5bf700 10 monclient(hunting): get_auth_request con 0x7fd4d80a08f0 auth_method 0 2023-09-30T01:54:37.671 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4de5bf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:37.671 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4de5bf700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:37.671 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4de5bf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:37.671 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4dddbe700 1 --2- 172.21.15.7:0/2920129776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4d809efd0 0x7fd4d809b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:37.672 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4de5bf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:37.672 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4de5bf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:37.672 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4de5bf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:37.672 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4dd5bd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:37.672 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.654+0000 7fd4dd5bd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:37.672 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.655+0000 7fd4dd5bd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:37.672 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.655+0000 7fd4dddbe700 10 monclient(hunting): get_auth_request con 0x7fd4d809efd0 auth_method 0 2023-09-30T01:54:37.672 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.655+0000 7fd4dddbe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:37.672 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.655+0000 7fd4dddbe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:37.672 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.655+0000 7fd4dddbe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:37.673 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.655+0000 7fd4dddbe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:37.673 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.655+0000 7fd4dddbe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:37.673 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.655+0000 7fd4dddbe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:37.673 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.655+0000 7fd4de5bf700 10 monclient(hunting): handle_auth_done global_id 4146 payload 824 2023-09-30T01:54:37.673 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.655+0000 7fd4de5bf700 1 -- 172.21.15.7:0/2920129776 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd4d809ffe0 msgr2=0x7fd4d809bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:37.673 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.655+0000 7fd4de5bf700 1 --2- 172.21.15.7:0/2920129776 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd4d809ffe0 0x7fd4d809bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:37.673 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.655+0000 7fd4de5bf700 1 -- 172.21.15.7:0/2920129776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4d809efd0 msgr2=0x7fd4d809b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:37.673 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.655+0000 7fd4de5bf700 1 --2- 172.21.15.7:0/2920129776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4d809efd0 0x7fd4d809b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:37.673 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.655+0000 7fd4de5bf700 10 monclient: _finish_hunting 0 2023-09-30T01:54:37.673 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.655+0000 7fd4de5bf700 1 monclient: found mon.noname-a 2023-09-30T01:54:37.674 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.655+0000 7fd4de5bf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:37.674 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.655+0000 7fd4de5bf700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:37.674 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.655+0000 7fd4de5bf700 1 -- 172.21.15.7:0/2920129776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d4009420 con 0x7fd4d80a08f0 2023-09-30T01:54:37.674 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4de5bf700 1 -- 172.21.15.7:0/2920129776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4d80a08f0 msgr2=0x7fd4d812c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:37.674 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4de5bf700 1 --2- 172.21.15.7:0/2920129776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4d80a08f0 0x7fd4d812c530 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd4d000cb60 tx=0x7fd4d0005340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:37.674 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4cdffb700 1 -- 172.21.15.7:0/2920129776 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d000a9f0 con 0x7fd4d80a08f0 2023-09-30T01:54:37.674 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4cdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:37.674 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4cdffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:37.674 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4cdffb700 10 monclient: dump: 2023-09-30T01:54:37.675 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-09-30T01:54:37.675 INFO:teuthology.misc.health.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:37.675 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:37.675 INFO:teuthology.misc.health.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:37.675 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:37.675 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:37.675 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:37.675 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:37.675 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:37.676 INFO:teuthology.misc.health.smithi007.stderr: 2023-09-30T01:54:37.676 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4cdffb700 10 monclient: _finish_auth 0 2023-09-30T01:54:37.676 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4cdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:37.676 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4cdffb700 1 -- 172.21.15.7:0/2920129776 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d000ab70 con 0x7fd4d80a08f0 2023-09-30T01:54:37.676 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4e0022700 5 monclient: authenticate success, global_id 4146 2023-09-30T01:54:37.676 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4e0022700 10 monclient: _renew_subs 2023-09-30T01:54:37.676 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4e0022700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:37.676 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4e0022700 1 -- 172.21.15.7:0/2920129776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c8009420 con 0x7fd4d80a08f0 2023-09-30T01:54:37.676 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4cdffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:37.676 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4cdffb700 1 -- 172.21.15.7:0/2920129776 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d00070f0 con 0x7fd4d80a08f0 2023-09-30T01:54:37.676 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4cdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:37.677 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4cdffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:37.677 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4cdffb700 10 monclient: dump: 2023-09-30T01:54:37.677 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-09-30T01:54:37.677 INFO:teuthology.misc.health.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:37.677 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:37.677 INFO:teuthology.misc.health.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:37.677 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:37.677 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:37.677 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:37.677 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:37.678 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:37.678 INFO:teuthology.misc.health.smithi007.stderr: 2023-09-30T01:54:37.678 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4cffff700 4 set_mon_vals no callback set 2023-09-30T01:54:37.678 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4e0022700 10 monclient: _renew_subs 2023-09-30T01:54:37.678 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4e0022700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:37.678 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4e0022700 1 -- 172.21.15.7:0/2920129776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d8130e20 con 0x7fd4d80a08f0 2023-09-30T01:54:37.678 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.657+0000 7fd4cdffb700 1 -- 172.21.15.7:0/2920129776 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fd4d00074a0 con 0x7fd4d80a08f0 2023-09-30T01:54:37.678 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.658+0000 7fd4cdffb700 1 -- 172.21.15.7:0/2920129776 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7fd4d001ba50 con 0x7fd4d80a08f0 2023-09-30T01:54:37.678 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.658+0000 7fd4e0022700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:37.678 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.658+0000 7fd4e0022700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:37.678 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.658+0000 7fd4e0022700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:37.679 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.658+0000 7fd4e0022700 1 -- 172.21.15.7:0/2920129776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4d805f230 con 0x7fd4d80a08f0 2023-09-30T01:54:37.679 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.660+0000 7fd4cdffb700 1 -- 172.21.15.7:0/2920129776 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fd4d0007930 con 0x7fd4d80a08f0 2023-09-30T01:54:37.679 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.661+0000 7fd4cdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:37.679 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.661+0000 7fd4cdffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:37.731 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.731+0000 7fd4e0022700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:54:37.731 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.731+0000 7fd4e0022700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:54:37.732 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.731+0000 7fd4e0022700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:37.732 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.731+0000 7fd4e0022700 1 -- 172.21.15.7:0/2920129776 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4d8068c20 con 0x7fd4d80a08f0 2023-09-30T01:54:37.732 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.731+0000 7fd4cdffb700 1 -- 172.21.15.7:0/2920129776 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5560 (secure 0 0 0) 0x7fd4d002a090 con 0x7fd4d80a08f0 2023-09-30T01:54:37.732 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.731+0000 7fd4cdffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:54:37.732 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.731+0000 7fd4cdffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:37.733 INFO:teuthology.misc.health.smithi007.stdout: 2023-09-30T01:54:37.733 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":8,"fsid":"2ea1044d-d984-49b9-baad-7f36bd454a19","created":"2023-09-30T01:54:34.356209+0000","modified":"2023-09-30T01:54:36.628886+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T01:54:36.628886+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"e44e9247-cb86-4a16-ac1f-f021cdf5108f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e30730fd-1e49-44bf-bdc9-087e23727df0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b39c552e-f5cc-4e4f-90a3-5aa5f5127a27","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"15cd89ae-049a-4e1a-8c6b-7d086ca6e1df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"de6773bd-d286-4ab9-b91f-8a90ee3073d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e6a847ab-276e-4d54-9a8f-6df5b8ab54cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:54:37.734 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.732+0000 7fd4e0022700 10 monclient: shutdown 2023-09-30T01:54:37.734 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.732+0000 7fd4e0022700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:37.734 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.732+0000 7fd4e0022700 1 -- 172.21.15.7:0/2920129776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4d80a08f0 msgr2=0x7fd4d812c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:37.734 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.732+0000 7fd4e0022700 1 --2- 172.21.15.7:0/2920129776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4d80a08f0 0x7fd4d812c530 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd4d000cb60 tx=0x7fd4d0005340 comp rx=0 tx=0).stop 2023-09-30T01:54:37.735 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.732+0000 7fd4e0022700 1 -- 172.21.15.7:0/2920129776 shutdown_connections 2023-09-30T01:54:37.735 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.732+0000 7fd4e0022700 1 --2- 172.21.15.7:0/2920129776 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4d80a08f0 0x7fd4d812c530 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:37.735 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.732+0000 7fd4e0022700 1 --2- 172.21.15.7:0/2920129776 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd4d809efd0 0x7fd4d809b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:37.735 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.732+0000 7fd4e0022700 1 --2- 172.21.15.7:0/2920129776 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd4d809ffe0 0x7fd4d809bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:37.735 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.732+0000 7fd4e0022700 1 -- 172.21.15.7:0/2920129776 >> 172.21.15.7:0/2920129776 conn(0x7fd4d80a55f0 msgr2=0x7fd4d8135b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:37.735 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.732+0000 7fd4e0022700 1 -- 172.21.15.7:0/2920129776 shutdown_connections 2023-09-30T01:54:37.736 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:37.732+0000 7fd4e0022700 1 -- 172.21.15.7:0/2920129776 wait complete. 2023-09-30T01:54:37.743 DEBUG:teuthology.misc:0 of 6 OSDs are up 2023-09-30T01:54:37.971 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:37.971+0000 7fc92c365180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-30T01:54:38.115 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:38.116+0000 7fc92c365180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T01:54:38.216 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:38.216+0000 7fc92c365180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T01:54:38.476 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:38.476+0000 7fc92c365180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-30T01:54:38.557 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:38.557+0000 7fc92c365180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T01:54:39.433 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:39.433+0000 7fc92c365180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-30T01:54:39.513 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:39.513+0000 7fc92c365180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T01:54:39.591 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:39.591+0000 7fc92c365180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T01:54:39.742 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:39.742+0000 7fc92c365180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T01:54:39.832 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:39.832+0000 7fc92c365180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T01:54:40.003 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:40.004+0000 7fc92c365180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T01:54:40.085 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:40.085+0000 7fc92c365180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T01:54:40.164 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:40.164+0000 7fc92c365180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T01:54:40.243 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:40.243+0000 7fc92c365180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T01:54:40.292 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:40.292+0000 7fc8bc657700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:54:40.292 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:40.292+0000 7fc8bc657700 -1 pg_autoscaler.serve: 2023-09-30T01:54:40.293 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:40.292+0000 7fc8bc657700 -1 Traceback (most recent call last): 2023-09-30T01:54:40.293 INFO:tasks.ceph.mgr.x.smithi050.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 325, in serve 2023-09-30T01:54:40.294 INFO:tasks.ceph.mgr.x.smithi050.stderr: self._maybe_adjust() 2023-09-30T01:54:40.294 INFO:tasks.ceph.mgr.x.smithi050.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 682, in _maybe_adjust 2023-09-30T01:54:40.294 INFO:tasks.ceph.mgr.x.smithi050.stderr: if self.noautoscale: 2023-09-30T01:54:40.294 INFO:tasks.ceph.mgr.x.smithi050.stderr:AttributeError: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T01:54:40.294 INFO:tasks.ceph.mgr.x.smithi050.stderr: 2023-09-30T01:54:40.507 INFO:tasks.ceph.osd.3.smithi050.stderr:2023-09-30T01:54:40.507+0000 7f7f853b0540 -1 osd.3 0 log_to_monitors true 2023-09-30T01:54:40.721 INFO:tasks.ceph.osd.5.smithi050.stderr:2023-09-30T01:54:40.722+0000 7fa51d706540 -1 osd.5 0 log_to_monitors true 2023-09-30T01:54:40.736 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-09-30T01:54:40.736+0000 7fcc43007540 -1 osd.2 0 log_to_monitors true 2023-09-30T01:54:40.751 INFO:tasks.ceph.osd.4.smithi050.stderr:2023-09-30T01:54:40.751+0000 7f24d6943540 -1 osd.4 0 log_to_monitors true 2023-09-30T01:54:40.772 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-09-30T01:54:40.772+0000 7f179de1b540 -1 osd.0 0 log_to_monitors true 2023-09-30T01:54:40.774 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-09-30T01:54:40.774+0000 7f35e7ff6540 -1 osd.1 0 log_to_monitors true 2023-09-30T01:54:41.270 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:41.270+0000 7fc8e5768700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-09-30T01:54:41.272 INFO:tasks.ceph.mgr.x.smithi050.stderr:2023-09-30T01:54:41.272+0000 7fc8e5768700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-09-30T01:54:42.357 INFO:tasks.ceph.osd.3.smithi050.stderr:2023-09-30T01:54:42.357+0000 7f7f7bfd1700 -1 osd.3 0 waiting for initial osdmap 2023-09-30T01:54:42.357 INFO:tasks.ceph.osd.5.smithi050.stderr:2023-09-30T01:54:42.357+0000 7fa514327700 -1 osd.5 0 waiting for initial osdmap 2023-09-30T01:54:42.358 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-09-30T01:54:42.357+0000 7f35dec17700 -1 osd.1 0 waiting for initial osdmap 2023-09-30T01:54:42.358 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-09-30T01:54:42.357+0000 7fcc39c28700 -1 osd.2 0 waiting for initial osdmap 2023-09-30T01:54:42.358 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-09-30T01:54:42.358+0000 7f1794a3c700 -1 osd.0 0 waiting for initial osdmap 2023-09-30T01:54:42.361 INFO:tasks.ceph.osd.3.smithi050.stderr:2023-09-30T01:54:42.362+0000 7f7f75dca700 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:54:42.362 INFO:tasks.ceph.osd.5.smithi050.stderr:2023-09-30T01:54:42.363+0000 7fa50d91f700 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:54:42.363 INFO:tasks.ceph.osd.1.smithi007.stderr:2023-09-30T01:54:42.363+0000 7f35d820f700 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:54:42.364 INFO:tasks.ceph.osd.0.smithi007.stderr:2023-09-30T01:54:42.364+0000 7f178e034700 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:54:42.368 INFO:tasks.ceph.osd.4.smithi050.stderr:2023-09-30T01:54:42.368+0000 7f24cd564700 -1 osd.4 0 waiting for initial osdmap 2023-09-30T01:54:42.376 INFO:tasks.ceph.osd.2.smithi007.stderr:2023-09-30T01:54:42.376+0000 7fcc33220700 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:54:42.854 INFO:tasks.ceph.osd.4.smithi050.stderr:2023-09-30T01:54:42.854+0000 7f24c6b5c700 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T01:54:44.349 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T01:54:44.377 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:54:44.377+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T01:54:44.469 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.469+0000 7ff5f4e89700 10 monclient: get_monmap_and_config 2023-09-30T01:54:44.470 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.469+0000 7ff5f4e89700 10 monclient: build_initial_monmap 2023-09-30T01:54:44.470 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.469+0000 7ff5f4e89700 10 monclient: monmap: 2023-09-30T01:54:44.470 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2023-09-30T01:54:44.470 INFO:teuthology.misc.health.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:44.471 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-09-30T01:54:44.470877+0000 2023-09-30T01:54:44.471 INFO:teuthology.misc.health.smithi007.stderr:created 2023-09-30T01:54:44.470877+0000 2023-09-30T01:54:44.471 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:44.471 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:44.471 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:44.471 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:44.471 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:44.471 INFO:teuthology.misc.health.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:44.471 INFO:teuthology.misc.health.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:44.472 INFO:teuthology.misc.health.smithi007.stderr: 2023-09-30T01:54:44.480 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.470+0000 7ff5f4e89700 1 Processor -- start 2023-09-30T01:54:44.481 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.470+0000 7ff5f4e89700 1 -- start start 2023-09-30T01:54:44.481 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.470+0000 7ff5f4e89700 10 monclient: init 2023-09-30T01:54:44.481 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.470+0000 7ff5f4e89700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:44.481 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5f4e89700 10 monclient: _add_conns ranks=[3,2,0,1,4] 2023-09-30T01:54:44.481 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5f4e89700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff5f009ffe0 0x7ff5f009ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:44.481 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5f4e89700 10 monclient(hunting): picked mon.noname-a con 0x7ff5f009ffe0 addr v2:172.21.15.7:3300/0 2023-09-30T01:54:44.481 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5f4e89700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f009efd0 0x7ff5f00a03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:44.481 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5f4e89700 10 monclient(hunting): picked mon.noname-b con 0x7ff5f009efd0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:44.481 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5f4e89700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff5f00a08f0 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:44.482 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5f4e89700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f0064fd0 con 0x7ff5f00a08f0 2023-09-30T01:54:44.482 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5f4e89700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:44.482 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5f4e89700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f00a1de0 con 0x7ff5f009ffe0 2023-09-30T01:54:44.482 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5f4e89700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:44.482 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5f4e89700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f009e040 con 0x7ff5f009efd0 2023-09-30T01:54:44.482 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5f4e89700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:44.482 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5f4e89700 10 monclient(hunting): authenticate will time out at 1173.536101s 2023-09-30T01:54:44.482 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5e7fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f009efd0 0x7ff5f00a03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:44.482 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5e7fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f009efd0 0x7ff5f00a03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59206/0 (socket says 172.21.15.7:59206) 2023-09-30T01:54:44.482 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5ee59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff5f009ffe0 0x7ff5f009ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:44.483 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5e7fff700 1 -- 172.21.15.7:0/1268277934 learned_addr learned my addr 172.21.15.7:0/1268277934 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:44.483 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5e7fff700 10 monclient(hunting): get_auth_request con 0x7ff5f009efd0 auth_method 0 2023-09-30T01:54:44.483 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5e7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:44.483 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5e7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:44.483 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5e7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:44.483 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5ee59c700 10 monclient(hunting): get_auth_request con 0x7ff5f009ffe0 auth_method 0 2023-09-30T01:54:44.483 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5ee59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:44.483 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5ee59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:44.483 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5ee59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:44.483 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5e7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:44.483 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5e7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:44.483 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5e7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:44.484 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5ee59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:44.484 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5ee59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:44.484 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5ee59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:44.484 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5e7fff700 10 monclient(hunting): handle_auth_done global_id 4184 payload 274 2023-09-30T01:54:44.484 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5e7fff700 1 -- 172.21.15.7:0/1268277934 >> v2:172.21.15.7:3300/0 conn(0x7ff5f009ffe0 msgr2=0x7ff5f009ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:44.484 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5e7fff700 1 --2- 172.21.15.7:0/1268277934 >> v2:172.21.15.7:3300/0 conn(0x7ff5f009ffe0 0x7ff5f009ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:44.484 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5e7fff700 1 -- 172.21.15.7:0/1268277934 >> v1:172.21.15.7:6789/0 conn(0x7ff5f00a08f0 legacy=0x7ff5f00984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:44.484 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5e7fff700 10 monclient: _finish_hunting 0 2023-09-30T01:54:44.484 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5e7fff700 1 monclient: found mon.noname-b 2023-09-30T01:54:44.484 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5e7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:44.484 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5e7fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:54:44.486 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5e7fff700 1 -- 172.21.15.7:0/1268277934 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f009da80 con 0x7ff5f009efd0 2023-09-30T01:54:44.486 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.471+0000 7ff5ee59c700 1 --2- 172.21.15.7:0/1268277934 >> v2:172.21.15.7:3300/0 conn(0x7ff5f009ffe0 0x7ff5f009ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:44.486 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.473+0000 7ff5e7fff700 1 -- 172.21.15.7:0/1268277934 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f009efd0 msgr2=0x7ff5f00a03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:44.486 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.473+0000 7ff5e7fff700 1 --2- 172.21.15.7:0/1268277934 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f009efd0 0x7ff5f00a03b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff5d80099b0 tx=0x7ff5d8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3a9ef6364a2e956 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:44.486 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5edd9b700 1 -- 172.21.15.7:0/1268277934 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8009ea0 con 0x7ff5f009efd0 2023-09-30T01:54:44.486 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5edd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:44.486 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5edd9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:54:44.487 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5edd9b700 10 monclient: dump: 2023-09-30T01:54:44.487 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-09-30T01:54:44.487 INFO:teuthology.misc.health.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:44.487 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:44.487 INFO:teuthology.misc.health.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:44.487 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:44.487 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:44.487 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:44.487 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:44.487 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:44.487 INFO:teuthology.misc.health.smithi007.stderr: 2023-09-30T01:54:44.488 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5edd9b700 10 monclient: _finish_auth 0 2023-09-30T01:54:44.488 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5edd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:44.488 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5edd9b700 1 -- 172.21.15.7:0/1268277934 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d800ba30 con 0x7ff5f009efd0 2023-09-30T01:54:44.488 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5f4e89700 5 monclient: authenticate success, global_id 4184 2023-09-30T01:54:44.488 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5f4e89700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:44.488 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5edd9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:44.488 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5f4e89700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:44.488 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5edd9b700 1 -- 172.21.15.7:0/1268277934 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d800bdd0 con 0x7ff5f009efd0 2023-09-30T01:54:44.488 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5edd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:44.488 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5f4e89700 4 set_mon_vals no callback set 2023-09-30T01:54:44.488 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5edd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:44.488 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5edd9b700 10 monclient: dump: 2023-09-30T01:54:44.489 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-09-30T01:54:44.489 INFO:teuthology.misc.health.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:44.489 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:44.489 INFO:teuthology.misc.health.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:44.489 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:44.489 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:44.489 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:44.489 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:44.489 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:44.489 INFO:teuthology.misc.health.smithi007.stderr: 2023-09-30T01:54:44.489 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5f4e89700 10 monclient: shutdown 2023-09-30T01:54:44.490 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5f4e89700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:44.490 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5f4e89700 1 -- 172.21.15.7:0/1268277934 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f009efd0 msgr2=0x7ff5f00a03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:44.490 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5f4e89700 1 --2- 172.21.15.7:0/1268277934 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f009efd0 0x7ff5f00a03b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff5d80099b0 tx=0x7ff5d8009250 comp rx=0 tx=0).stop 2023-09-30T01:54:44.490 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5f4e89700 1 -- 172.21.15.7:0/1268277934 shutdown_connections 2023-09-30T01:54:44.490 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5f4e89700 1 --2- 172.21.15.7:0/1268277934 >> v2:172.21.15.7:3300/0 conn(0x7ff5f009ffe0 0x7ff5f009ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:44.490 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5f4e89700 1 --2- 172.21.15.7:0/1268277934 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f009efd0 0x7ff5f00a03b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:44.490 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5f4e89700 1 -- 172.21.15.7:0/1268277934 >> 172.21.15.7:0/1268277934 conn(0x7ff5f00a55f0 msgr2=0x7ff5f00a59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:44.490 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5f4e89700 1 -- 172.21.15.7:0/1268277934 shutdown_connections 2023-09-30T01:54:44.490 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.474+0000 7ff5f4e89700 1 -- 172.21.15.7:0/1268277934 wait complete. 2023-09-30T01:54:44.490 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.475+0000 7ff5f4e89700 10 monclient: build_initial_monmap 2023-09-30T01:54:44.490 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.475+0000 7ff5f4e89700 10 monclient: monmap: 2023-09-30T01:54:44.491 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2023-09-30T01:54:44.491 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:44.491 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:44.491 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2023-09-30T01:54:44.491 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:44.491 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:44.491 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:44.491 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:44.491 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:44.491 INFO:teuthology.misc.health.smithi007.stderr: 2023-09-30T01:54:44.491 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.475+0000 7ff5f4e89700 1 Processor -- start 2023-09-30T01:54:44.491 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.475+0000 7ff5f4e89700 1 -- start start 2023-09-30T01:54:44.492 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.475+0000 7ff5f4e89700 10 monclient: init 2023-09-30T01:54:44.492 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5f4e89700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:44.492 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5f4e89700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:54:44.492 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5f4e89700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5f009efd0 0x7ff5f009b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:44.492 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5f4e89700 10 monclient(hunting): picked mon.noname-a con 0x7ff5f009efd0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:44.492 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5f4e89700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f009ffe0 0x7ff5f009ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:44.492 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5f4e89700 10 monclient(hunting): picked mon.noname-c con 0x7ff5f009ffe0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:44.492 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5f4e89700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5f00a08f0 0x7ff5f009bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:44.492 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5f4e89700 10 monclient(hunting): picked mon.noname-b con 0x7ff5f00a08f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:44.492 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5f4e89700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:44.492 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5f4e89700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f0133960 con 0x7ff5f009efd0 2023-09-30T01:54:44.492 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5f4e89700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:44.493 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5f4e89700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f006aea0 con 0x7ff5f00a08f0 2023-09-30T01:54:44.493 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5f4e89700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:44.493 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5f4e89700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f012dda0 con 0x7ff5f009ffe0 2023-09-30T01:54:44.493 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5f4e89700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:44.493 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5f4e89700 10 monclient(hunting): authenticate will time out at 1173.541127s 2023-09-30T01:54:44.493 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5e7fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f009ffe0 0x7ff5f009ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:44.493 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5ee59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5f009efd0 0x7ff5f009b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:44.493 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5e7fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f009ffe0 0x7ff5f009ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59208/0 (socket says 172.21.15.7:59208) 2023-09-30T01:54:44.493 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5e7fff700 1 -- 172.21.15.7:0/948565960 learned_addr learned my addr 172.21.15.7:0/948565960 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:44.493 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5e7fff700 10 monclient(hunting): get_auth_request con 0x7ff5f009ffe0 auth_method 0 2023-09-30T01:54:44.493 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5e7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:44.493 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5e7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:44.494 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5e7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:44.494 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5ee59c700 10 monclient(hunting): get_auth_request con 0x7ff5f009efd0 auth_method 0 2023-09-30T01:54:44.494 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5ee59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:44.494 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5ee59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:44.494 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5ee59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:44.494 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5eed9d700 1 --2- 172.21.15.7:0/948565960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5f00a08f0 0x7ff5f009bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:44.494 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5e7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:44.494 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5e7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:44.494 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5e7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:44.494 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5eed9d700 10 monclient(hunting): get_auth_request con 0x7ff5f00a08f0 auth_method 0 2023-09-30T01:54:44.494 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5eed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:44.495 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5eed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:44.495 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5eed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:44.495 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5ee59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:44.495 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5ee59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:44.495 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5ee59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:44.495 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5eed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:44.495 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5eed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:44.495 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5eed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:44.495 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5e7fff700 10 monclient(hunting): handle_auth_done global_id 4187 payload 824 2023-09-30T01:54:44.495 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5e7fff700 1 -- 172.21.15.7:0/948565960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5f00a08f0 msgr2=0x7ff5f009bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:44.495 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5e7fff700 1 --2- 172.21.15.7:0/948565960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5f00a08f0 0x7ff5f009bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:44.496 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5e7fff700 1 -- 172.21.15.7:0/948565960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5f009efd0 msgr2=0x7ff5f009b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:44.496 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5e7fff700 1 --2- 172.21.15.7:0/948565960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5f009efd0 0x7ff5f009b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:44.496 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5e7fff700 10 monclient: _finish_hunting 0 2023-09-30T01:54:44.496 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5e7fff700 1 monclient: found mon.noname-c 2023-09-30T01:54:44.496 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5e7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:44.496 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5e7fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:54:44.496 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.476+0000 7ff5e7fff700 1 -- 172.21.15.7:0/948565960 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e0009420 con 0x7ff5f009ffe0 2023-09-30T01:54:44.496 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.478+0000 7ff5e7fff700 1 -- 172.21.15.7:0/948565960 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f009ffe0 msgr2=0x7ff5f009ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:44.496 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.478+0000 7ff5e7fff700 1 --2- 172.21.15.7:0/948565960 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f009ffe0 0x7ff5f009ba20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00a1570 tx=0x7ff5d8016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:44.496 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.478+0000 7ff5e67fc700 1 -- 172.21.15.7:0/948565960 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d801b470 con 0x7ff5f009ffe0 2023-09-30T01:54:44.496 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.478+0000 7ff5e67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:44.497 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.478+0000 7ff5e67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:44.497 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.478+0000 7ff5e67fc700 10 monclient: dump: 2023-09-30T01:54:44.497 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-09-30T01:54:44.497 INFO:teuthology.misc.health.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:44.497 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:44.497 INFO:teuthology.misc.health.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:44.497 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:44.497 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:44.497 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:44.497 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:44.497 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:44.497 INFO:teuthology.misc.health.smithi007.stderr: 2023-09-30T01:54:44.497 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.478+0000 7ff5e67fc700 10 monclient: _finish_auth 0 2023-09-30T01:54:44.498 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.478+0000 7ff5e67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:44.498 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.478+0000 7ff5e67fc700 1 -- 172.21.15.7:0/948565960 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d801b5f0 con 0x7ff5f009ffe0 2023-09-30T01:54:44.498 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.478+0000 7ff5f4e89700 5 monclient: authenticate success, global_id 4187 2023-09-30T01:54:44.498 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.478+0000 7ff5f4e89700 10 monclient: _renew_subs 2023-09-30T01:54:44.498 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.478+0000 7ff5f4e89700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:44.498 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.478+0000 7ff5f4e89700 1 -- 172.21.15.7:0/948565960 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5d80094f0 con 0x7ff5f009ffe0 2023-09-30T01:54:44.498 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.479+0000 7ff5e67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:44.498 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.479+0000 7ff5e67fc700 1 -- 172.21.15.7:0/948565960 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d801b9b0 con 0x7ff5f009ffe0 2023-09-30T01:54:44.498 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.479+0000 7ff5e67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:44.498 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.479+0000 7ff5e67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:44.498 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.479+0000 7ff5e67fc700 10 monclient: dump: 2023-09-30T01:54:44.499 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2023-09-30T01:54:44.499 INFO:teuthology.misc.health.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:44.499 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:44.499 INFO:teuthology.misc.health.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:44.499 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:44.499 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:44.499 INFO:teuthology.misc.health.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:44.499 INFO:teuthology.misc.health.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:44.499 INFO:teuthology.misc.health.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:44.499 INFO:teuthology.misc.health.smithi007.stderr: 2023-09-30T01:54:44.499 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.479+0000 7ff5ed59a700 4 set_mon_vals no callback set 2023-09-30T01:54:44.499 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.479+0000 7ff5f4e89700 10 monclient: _renew_subs 2023-09-30T01:54:44.500 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.479+0000 7ff5f4e89700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:44.500 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.479+0000 7ff5f4e89700 1 -- 172.21.15.7:0/948565960 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f012e370 con 0x7ff5f009ffe0 2023-09-30T01:54:44.500 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.479+0000 7ff5e67fc700 1 -- 172.21.15.7:0/948565960 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 4) v1 ==== 42485+0+0 (secure 0 0 0) 0x7ff5d801c450 con 0x7ff5f009ffe0 2023-09-30T01:54:44.500 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.479+0000 7ff5f4e89700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:44.500 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.479+0000 7ff5f4e89700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:44.500 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.479+0000 7ff5f4e89700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:44.500 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.479+0000 7ff5f4e89700 1 -- 172.21.15.7:0/948565960 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f005f230 con 0x7ff5f009ffe0 2023-09-30T01:54:44.500 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.480+0000 7ff5e67fc700 1 --2- 172.21.15.7:0/948565960 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff5d00357a0 0x7ff5d0037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:44.500 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.480+0000 7ff5e67fc700 1 -- 172.21.15.7:0/948565960 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 3603+0+0 (secure 0 0 0) 0x7ff5d8049340 con 0x7ff5f009ffe0 2023-09-30T01:54:44.500 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.480+0000 7ff5ee59c700 1 --2- 172.21.15.7:0/948565960 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff5d00357a0 0x7ff5d0037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:44.500 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.480+0000 7ff5ee59c700 10 monclient: get_auth_request con 0x7ff5d00357a0 auth_method 0 2023-09-30T01:54:44.501 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.482+0000 7ff5ee59c700 1 --2- 172.21.15.7:0/948565960 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff5d00357a0 0x7ff5d0037c50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0009f40 tx=0x7ff5e00091a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:44.501 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.485+0000 7ff5e67fc700 1 -- 172.21.15.7:0/948565960 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff5d8018760 con 0x7ff5f009ffe0 2023-09-30T01:54:44.501 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.485+0000 7ff5e67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:44.501 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.485+0000 7ff5e67fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:44.681 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.681+0000 7ff5f4e89700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:54:44.682 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.681+0000 7ff5f4e89700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:54:44.682 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.681+0000 7ff5f4e89700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:44.682 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.681+0000 7ff5f4e89700 1 -- 172.21.15.7:0/948565960 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5f0068c20 con 0x7ff5f009ffe0 2023-09-30T01:54:44.682 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.682+0000 7ff5e67fc700 1 -- 172.21.15.7:0/948565960 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+8032 (secure 0 0 0) 0x7ff5d801a3d0 con 0x7ff5f009ffe0 2023-09-30T01:54:44.682 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.682+0000 7ff5e67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:54:44.683 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.682+0000 7ff5e67fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:44.683 INFO:teuthology.misc.health.smithi007.stdout: 2023-09-30T01:54:44.683 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":11,"fsid":"2ea1044d-d984-49b9-baad-7f36bd454a19","created":"2023-09-30T01:54:34.356209+0000","modified":"2023-09-30T01:54:43.359934+0000","last_up_change":"2023-09-30T01:54:43.359934+0000","last_in_change":"2023-09-30T01:54:36.628886+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"e44e9247-cb86-4a16-ac1f-f021cdf5108f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":86601},{"type":"v1","addr":"172.21.15.7:6801","nonce":86601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":86601},{"type":"v1","addr":"172.21.15.7:6803","nonce":86601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":86601},{"type":"v1","addr":"172.21.15.7:6807","nonce":86601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":86601},{"type":"v1","addr":"172.21.15.7:6805","nonce":86601}]},"public_addr":"172.21.15.7:6801/86601","cluster_addr":"172.21.15.7:6803/86601","heartbeat_back_addr":"172.21.15.7:6807/86601","heartbeat_front_addr":"172.21.15.7:6805/86601","state":["exists","up"]},{"osd":1,"uuid":"e30730fd-1e49-44bf-bdc9-087e23727df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":86600},{"type":"v1","addr":"172.21.15.7:6817","nonce":86600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":86600},{"type":"v1","addr":"172.21.15.7:6819","nonce":86600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":86600},{"type":"v1","addr":"172.21.15.7:6823","nonce":86600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":86600},{"type":"v1","addr":"172.21.15.7:6821","nonce":86600}]},"public_addr":"172.21.15.7:6817/86600","cluster_addr":"172.21.15.7:6819/86600","heartbeat_back_addr":"172.21.15.7:6823/86600","heartbeat_front_addr":"172.21.15.7:6821/86600","state":["exists","up"]},{"osd":2,"uuid":"b39c552e-f5cc-4e4f-90a3-5aa5f5127a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":86602},{"type":"v1","addr":"172.21.15.7:6809","nonce":86602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":86602},{"type":"v1","addr":"172.21.15.7:6811","nonce":86602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":86602},{"type":"v1","addr":"172.21.15.7:6815","nonce":86602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":86602},{"type":"v1","addr":"172.21.15.7:6813","nonce":86602}]},"public_addr":"172.21.15.7:6809/86602","cluster_addr":"172.21.15.7:6811/86602","heartbeat_back_addr":"172.21.15.7:6815/86602","heartbeat_front_addr":"172.21.15.7:6813/86602","state":["exists","up"]},{"osd":3,"uuid":"15cd89ae-049a-4e1a-8c6b-7d086ca6e1df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":86633},{"type":"v1","addr":"172.21.15.50:6807","nonce":86633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":86633},{"type":"v1","addr":"172.21.15.50:6812","nonce":86633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6819","nonce":86633},{"type":"v1","addr":"172.21.15.50:6821","nonce":86633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6815","nonce":86633},{"type":"v1","addr":"172.21.15.50:6818","nonce":86633}]},"public_addr":"172.21.15.50:6807/86633","cluster_addr":"172.21.15.50:6812/86633","heartbeat_back_addr":"172.21.15.50:6821/86633","heartbeat_front_addr":"172.21.15.50:6818/86633","state":["exists","up"]},{"osd":4,"uuid":"de6773bd-d286-4ab9-b91f-8a90ee3073d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":86632},{"type":"v1","addr":"172.21.15.50:6806","nonce":86632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":86632},{"type":"v1","addr":"172.21.15.50:6814","nonce":86632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":86632},{"type":"v1","addr":"172.21.15.50:6823","nonce":86632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":86632},{"type":"v1","addr":"172.21.15.50:6820","nonce":86632}]},"public_addr":"172.21.15.50:6806/86632","cluster_addr":"172.21.15.50:6814/86632","heartbeat_back_addr":"172.21.15.50:6823/86632","heartbeat_front_addr":"172.21.15.50:6820/86632","state":["exists","up"]},{"osd":5,"uuid":"e6a847ab-276e-4d54-9a8f-6df5b8ab54cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":86634},{"type":"v1","addr":"172.21.15.50:6801","nonce":86634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":86634},{"type":"v1","addr":"172.21.15.50:6805","nonce":86634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":86634},{"type":"v1","addr":"172.21.15.50:6816","nonce":86634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":86634},{"type":"v1","addr":"172.21.15.50:6811","nonce":86634}]},"public_addr":"172.21.15.50:6801/86634","cluster_addr":"172.21.15.50:6805/86634","heartbeat_back_addr":"172.21.15.50:6816/86634","heartbeat_front_addr":"172.21.15.50:6811/86634","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:54:44.685 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.684+0000 7ff5f4e89700 1 -- 172.21.15.7:0/948565960 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff5d00357a0 msgr2=0x7ff5d0037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:44.685 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.684+0000 7ff5f4e89700 1 --2- 172.21.15.7:0/948565960 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff5d00357a0 0x7ff5d0037c50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0009f40 tx=0x7ff5e00091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:44.685 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.684+0000 7ff5f4e89700 10 monclient: shutdown 2023-09-30T01:54:44.685 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.684+0000 7ff5f4e89700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:44.685 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.684+0000 7ff5f4e89700 1 -- 172.21.15.7:0/948565960 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f009ffe0 msgr2=0x7ff5f009ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:44.685 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.684+0000 7ff5f4e89700 1 --2- 172.21.15.7:0/948565960 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f009ffe0 0x7ff5f009ba20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00a1570 tx=0x7ff5d8016000 comp rx=0 tx=0).stop 2023-09-30T01:54:44.685 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.684+0000 7ff5f4e89700 1 -- 172.21.15.7:0/948565960 shutdown_connections 2023-09-30T01:54:44.685 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.684+0000 7ff5f4e89700 1 --2- 172.21.15.7:0/948565960 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff5d00357a0 0x7ff5d0037c50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:44.685 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.684+0000 7ff5f4e89700 1 --2- 172.21.15.7:0/948565960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5f009efd0 0x7ff5f009b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:44.685 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.684+0000 7ff5f4e89700 1 --2- 172.21.15.7:0/948565960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5f00a08f0 0x7ff5f009bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:44.686 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.684+0000 7ff5f4e89700 1 --2- 172.21.15.7:0/948565960 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5f009ffe0 0x7ff5f009ba20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:44.686 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.684+0000 7ff5f4e89700 1 -- 172.21.15.7:0/948565960 >> 172.21.15.7:0/948565960 conn(0x7ff5f00a55f0 msgr2=0x7ff5f0135b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:44.686 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.684+0000 7ff5f4e89700 1 -- 172.21.15.7:0/948565960 shutdown_connections 2023-09-30T01:54:44.686 INFO:teuthology.misc.health.smithi007.stderr:2023-09-30T01:54:44.684+0000 7ff5f4e89700 1 -- 172.21.15.7:0/948565960 wait complete. 2023-09-30T01:54:44.695 DEBUG:teuthology.misc:6 of 6 OSDs are up 2023-09-30T01:54:44.695 INFO:tasks.ceph:Creating RBD pool 2023-09-30T01:54:44.695 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-30T01:54:44.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.852+0000 7f16d0540700 10 monclient: get_monmap_and_config 2023-09-30T01:54:44.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.852+0000 7f16d0540700 10 monclient: build_initial_monmap 2023-09-30T01:54:44.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.852+0000 7f16d0540700 10 monclient: monmap: 2023-09-30T01:54:44.852 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:44.853 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:44.853 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:44.853516+0000 2023-09-30T01:54:44.853 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:44.853516+0000 2023-09-30T01:54:44.853 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:44.853 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:44.853 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:44.853 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:44.853 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:44.853 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:44.853 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:44.853 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:44.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16d0540700 1 Processor -- start 2023-09-30T01:54:44.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16d0540700 1 -- start start 2023-09-30T01:54:44.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16d0540700 10 monclient: init 2023-09-30T01:54:44.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16d0540700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:44.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16d0540700 10 monclient: _add_conns ranks=[2,3,0,4,1] 2023-09-30T01:54:44.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16d0540700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16c8096a50 0x7f16c8132680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:44.860 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16d0540700 10 monclient(hunting): picked mon.noname-b con 0x7f16c8096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:44.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16d0540700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f16c8096e20 0x7f16c80971f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:44.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16d0540700 10 monclient(hunting): picked mon.noname-a con 0x7f16c8096e20 addr v2:172.21.15.7:3300/0 2023-09-30T01:54:44.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16d0540700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f16c812ef00 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:44.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16d0540700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c8099440 con 0x7f16c812ef00 2023-09-30T01:54:44.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16d0540700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:44.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16d0540700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16c8097b40 con 0x7f16c8096e20 2023-09-30T01:54:44.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16d0540700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:44.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16d0540700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c8099160 con 0x7f16c8096a50 2023-09-30T01:54:44.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16d0540700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:44.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16d0540700 10 monclient(hunting): authenticate will time out at 1173.918696s 2023-09-30T01:54:44.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16ce2dc700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16c8096a50 0x7f16c8132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:44.861 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16cdadb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f16c8096e20 0x7f16c80971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:44.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16ceadd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f16c812ef00 0x7f16c812f370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58052/0 (socket says 172.21.15.7:58052) 2023-09-30T01:54:44.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16ceadd700 1 -- 172.21.15.7:0/3202169575 learned_addr learned my addr 172.21.15.7:0/3202169575 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:54:44.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16ce2dc700 1 --2- 172.21.15.7:0/3202169575 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16c8096a50 0x7f16c8132680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59224/0 (socket says 172.21.15.7:59224) 2023-09-30T01:54:44.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16ce2dc700 10 monclient(hunting): get_auth_request con 0x7f16c8096a50 auth_method 0 2023-09-30T01:54:44.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16ce2dc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:44.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16ce2dc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:44.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16ce2dc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:44.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16cdadb700 10 monclient(hunting): get_auth_request con 0x7f16c8096e20 auth_method 0 2023-09-30T01:54:44.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16cdadb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:44.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16cdadb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:44.862 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16cdadb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:44.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16ce2dc700 1 -- 172.21.15.7:0/3202169575 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16c8096a50 msgr2=0x7f16c8132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-30T01:54:44.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16ce2dc700 1 -- 172.21.15.7:0/3202169575 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16c8096a50 msgr2=0x7f16c8132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:54:44.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16ce2dc700 1 --2- 172.21.15.7:0/3202169575 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16c8096a50 0x7f16c8132680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:54:44.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16ce2dc700 1 --2- 172.21.15.7:0/3202169575 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16c8096a50 0x7f16c8132680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:54:44.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16cdadb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:44.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.853+0000 7f16cdadb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:44.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.854+0000 7f16cdadb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:44.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.854+0000 7f16cdadb700 10 monclient(hunting): handle_auth_done global_id 4191 payload 274 2023-09-30T01:54:44.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.854+0000 7f16cdadb700 1 -- 172.21.15.7:0/3202169575 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16c8096a50 msgr2=0x7f16c8132680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:54:44.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.854+0000 7f16cdadb700 1 --2- 172.21.15.7:0/3202169575 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16c8096a50 0x7f16c8132680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:44.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.854+0000 7f16cdadb700 1 -- 172.21.15.7:0/3202169575 >> v1:172.21.15.7:6789/0 conn(0x7f16c812ef00 legacy=0x7f16c812f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:44.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.854+0000 7f16cdadb700 10 monclient: _finish_hunting 0 2023-09-30T01:54:44.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.854+0000 7f16cdadb700 1 monclient: found mon.noname-a 2023-09-30T01:54:44.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.854+0000 7f16cdadb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:44.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.854+0000 7f16cdadb700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:44.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.854+0000 7f16cdadb700 1 -- 172.21.15.7:0/3202169575 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c8099720 con 0x7f16c8096e20 2023-09-30T01:54:44.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16cdadb700 1 -- 172.21.15.7:0/3202169575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16c8096e20 msgr2=0x7f16c80971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:44.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16cdadb700 1 --2- 172.21.15.7:0/3202169575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16c8096e20 0x7f16c80971f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f16bc0098e0 tx=0x7f16bc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=852068a610ed8615 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:44.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16cd2da700 1 -- 172.21.15.7:0/3202169575 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16bc009e80 con 0x7f16c8096e20 2023-09-30T01:54:44.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16cd2da700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:44.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16cd2da700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:44.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16cd2da700 10 monclient: dump: 2023-09-30T01:54:44.864 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:44.864 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:44.865 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:44.865 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:44.865 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:44.865 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:44.865 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:44.865 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:44.865 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:44.865 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:44.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16cd2da700 10 monclient: _finish_auth 0 2023-09-30T01:54:44.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16cd2da700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:44.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16d0540700 5 monclient: authenticate success, global_id 4191 2023-09-30T01:54:44.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16cd2da700 1 -- 172.21.15.7:0/3202169575 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16bc00ba30 con 0x7f16c8096e20 2023-09-30T01:54:44.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16d0540700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:44.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16cd2da700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:44.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16d0540700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:44.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16d0540700 4 set_mon_vals no callback set 2023-09-30T01:54:44.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16cd2da700 1 -- 172.21.15.7:0/3202169575 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16bc00bdb0 con 0x7f16c8096e20 2023-09-30T01:54:44.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16cd2da700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:44.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16cd2da700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:44.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16cd2da700 10 monclient: dump: 2023-09-30T01:54:44.866 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:44.866 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:44.866 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:44.866 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:44.867 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:44.867 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:44.867 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:44.867 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:44.867 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:44.867 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:44.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16d0540700 10 monclient: shutdown 2023-09-30T01:54:44.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16d0540700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:44.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16d0540700 1 -- 172.21.15.7:0/3202169575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16c8096e20 msgr2=0x7f16c80971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:44.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.856+0000 7f16d0540700 1 --2- 172.21.15.7:0/3202169575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16c8096e20 0x7f16c80971f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f16bc0098e0 tx=0x7f16bc0091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:44.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.857+0000 7f16d0540700 1 -- 172.21.15.7:0/3202169575 shutdown_connections 2023-09-30T01:54:44.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.857+0000 7f16d0540700 1 --2- 172.21.15.7:0/3202169575 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16c8096a50 0x7f16c8132680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:44.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.857+0000 7f16d0540700 1 --2- 172.21.15.7:0/3202169575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16c8096e20 0x7f16c80971f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:44.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.857+0000 7f16d0540700 1 -- 172.21.15.7:0/3202169575 >> 172.21.15.7:0/3202169575 conn(0x7f16c8070cf0 msgr2=0x7f16c80710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:44.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.857+0000 7f16d0540700 1 -- 172.21.15.7:0/3202169575 shutdown_connections 2023-09-30T01:54:44.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.857+0000 7f16d0540700 1 -- 172.21.15.7:0/3202169575 wait complete. 2023-09-30T01:54:44.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.857+0000 7f16d0540700 10 monclient: build_initial_monmap 2023-09-30T01:54:44.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.857+0000 7f16d0540700 10 monclient: monmap: 2023-09-30T01:54:44.868 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:44.868 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:44.868 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:44.868 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:44.868 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:44.869 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:44.869 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:44.869 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:44.869 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:44.869 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:44.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.857+0000 7f16d0540700 1 Processor -- start 2023-09-30T01:54:44.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16d0540700 1 -- start start 2023-09-30T01:54:44.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16d0540700 10 monclient: init 2023-09-30T01:54:44.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16d0540700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:44.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16d0540700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:54:44.869 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16d0540700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16c8096a50 0x7f16c812d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:44.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16d0540700 10 monclient(hunting): picked mon.noname-c con 0x7f16c8096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:44.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16d0540700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16c8096e20 0x7f16c812da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:44.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16d0540700 10 monclient(hunting): picked mon.noname-a con 0x7f16c8096e20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:44.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16d0540700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16c812ef00 0x7f16c812dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:44.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16d0540700 10 monclient(hunting): picked mon.noname-b con 0x7f16c812ef00 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:44.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16d0540700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:44.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16d0540700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c80a15f0 con 0x7f16c8096e20 2023-09-30T01:54:44.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16d0540700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:44.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16d0540700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c80a4ac0 con 0x7f16c812ef00 2023-09-30T01:54:44.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16d0540700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:44.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16d0540700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c80a1310 con 0x7f16c8096a50 2023-09-30T01:54:44.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16d0540700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:44.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16d0540700 10 monclient(hunting): authenticate will time out at 1173.923489s 2023-09-30T01:54:44.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16ce2dc700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16c8096a50 0x7f16c812d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:44.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16cdadb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16c8096e20 0x7f16c812da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:44.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16cdadb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16c8096e20 0x7f16c812da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54618/0 (socket says 172.21.15.7:54618) 2023-09-30T01:54:44.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16cdadb700 1 -- 172.21.15.7:0/1900425101 learned_addr learned my addr 172.21.15.7:0/1900425101 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:44.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16ce2dc700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16c8096a50 0x7f16c812d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59238/0 (socket says 172.21.15.7:59238) 2023-09-30T01:54:44.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16cdadb700 10 monclient(hunting): get_auth_request con 0x7f16c8096e20 auth_method 0 2023-09-30T01:54:44.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16cdadb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:44.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16cdadb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:44.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16cdadb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:44.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16ce2dc700 10 monclient(hunting): get_auth_request con 0x7f16c8096a50 auth_method 0 2023-09-30T01:54:44.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16ce2dc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:44.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16ce2dc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:44.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16ce2dc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:44.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16cdadb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:44.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16cdadb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:44.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16cdadb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:44.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16ce2dc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:44.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16ce2dc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:44.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16ce2dc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:44.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16cdadb700 10 monclient(hunting): handle_auth_done global_id 4194 payload 824 2023-09-30T01:54:44.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16ceadd700 1 --2- 172.21.15.7:0/1900425101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16c812ef00 0x7f16c812dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:44.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16cdadb700 1 -- 172.21.15.7:0/1900425101 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16c8096a50 msgr2=0x7f16c812d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:44.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16cdadb700 1 --2- 172.21.15.7:0/1900425101 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16c8096a50 0x7f16c812d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:44.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16cdadb700 1 -- 172.21.15.7:0/1900425101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16c812ef00 msgr2=0x7f16c812dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:44.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16cdadb700 1 --2- 172.21.15.7:0/1900425101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16c812ef00 0x7f16c812dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:44.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16cdadb700 10 monclient: _finish_hunting 0 2023-09-30T01:54:44.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16cdadb700 1 monclient: found mon.noname-a 2023-09-30T01:54:44.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16cdadb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:44.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16cdadb700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:44.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.858+0000 7f16cdadb700 1 -- 172.21.15.7:0/1900425101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc009420 con 0x7f16c8096e20 2023-09-30T01:54:44.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16cdadb700 1 -- 172.21.15.7:0/1900425101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16c8096e20 msgr2=0x7f16c812da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:44.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16cdadb700 1 --2- 172.21.15.7:0/1900425101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16c8096e20 0x7f16c812da80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f16c80a2b60 tx=0x7f16bc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:44.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16c67fc700 1 -- 172.21.15.7:0/1900425101 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16bc006af0 con 0x7f16c8096e20 2023-09-30T01:54:44.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:44.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16c67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:44.876 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16c67fc700 10 monclient: dump: 2023-09-30T01:54:44.876 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:44.876 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:44.876 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:44.876 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:44.877 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:44.877 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:44.877 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:44.877 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:44.877 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:44.877 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:44.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16c67fc700 10 monclient: _finish_auth 0 2023-09-30T01:54:44.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16c67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:44.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16d0540700 5 monclient: authenticate success, global_id 4194 2023-09-30T01:54:44.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16c67fc700 1 -- 172.21.15.7:0/1900425101 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16bc020070 con 0x7f16c8096e20 2023-09-30T01:54:44.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16d0540700 10 monclient: _renew_subs 2023-09-30T01:54:44.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16d0540700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:44.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16d0540700 1 -- 172.21.15.7:0/1900425101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c80a0140 con 0x7f16c8096e20 2023-09-30T01:54:44.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16c67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:44.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16c67fc700 1 -- 172.21.15.7:0/1900425101 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16bc000c90 con 0x7f16c8096e20 2023-09-30T01:54:44.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:44.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16c67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:44.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16ccad9700 4 set_mon_vals no callback set 2023-09-30T01:54:44.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16c67fc700 10 monclient: dump: 2023-09-30T01:54:44.878 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:44.878 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:44.878 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:44.878 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:44.878 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:44.879 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:44.879 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:44.879 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:44.879 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:44.879 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:44.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16d0540700 10 monclient: _renew_subs 2023-09-30T01:54:44.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16d0540700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:44.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.861+0000 7f16d0540700 1 -- 172.21.15.7:0/1900425101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c809f500 con 0x7f16c8096e20 2023-09-30T01:54:44.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.862+0000 7f16c67fc700 1 -- 172.21.15.7:0/1900425101 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 4) v1 ==== 42485+0+0 (secure 0 0 0) 0x7f16bc022030 con 0x7f16c8096e20 2023-09-30T01:54:44.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.862+0000 7f16d0540700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:44.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.862+0000 7f16d0540700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:44.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.862+0000 7f16d0540700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:44.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.862+0000 7f16d0540700 1 -- 172.21.15.7:0/1900425101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c8064c10 con 0x7f16c8096e20 2023-09-30T01:54:44.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.862+0000 7f16c67fc700 1 --2- 172.21.15.7:0/1900425101 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f16b40357d0 0x7f16b4037c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:44.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.862+0000 7f16c67fc700 1 -- 172.21.15.7:0/1900425101 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 3603+0+0 (secure 0 0 0) 0x7f16bc0495c0 con 0x7f16c8096e20 2023-09-30T01:54:44.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.862+0000 7f16ce2dc700 1 --2- 172.21.15.7:0/1900425101 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f16b40357d0 0x7f16b4037c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:44.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.862+0000 7f16ce2dc700 10 monclient: get_auth_request con 0x7f16b40357d0 auth_method 0 2023-09-30T01:54:44.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.865+0000 7f16ce2dc700 1 --2- 172.21.15.7:0/1900425101 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f16b40357d0 0x7f16b4037c80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f16b800afd0 tx=0x7f16b80098b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:44.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.867+0000 7f16c67fc700 1 -- 172.21.15.7:0/1900425101 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f16bc01b660 con 0x7f16c8096e20 2023-09-30T01:54:44.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.867+0000 7f16c67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:44.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:44.867+0000 7f16c67fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:45.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:45.060+0000 7f16d0540700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-09-30T01:54:45.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:45.060+0000 7f16d0540700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-09-30T01:54:45.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:45.061+0000 7f16d0540700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:45.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:45.061+0000 7f16d0540700 1 -- 172.21.15.7:0/1900425101 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f16c8064c10 con 0x7f16c8096e20 2023-09-30T01:54:45.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:45.892+0000 7f16c67fc700 1 -- 172.21.15.7:0/1900425101 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) v1 ==== 109+0+0 (secure 0 0 0) 0x7f16bc0193f0 con 0x7f16c8096e20 2023-09-30T01:54:45.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:45.892+0000 7f16c67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-09-30T01:54:45.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:45.892+0000 7f16c67fc700 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2023-09-30T01:54:45.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:45.897+0000 7f16d0540700 1 -- 172.21.15.7:0/1900425101 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f16b40357d0 msgr2=0x7f16b4037c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:45.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:45.897+0000 7f16d0540700 1 --2- 172.21.15.7:0/1900425101 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f16b40357d0 0x7f16b4037c80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f16b800afd0 tx=0x7f16b80098b0 comp rx=0 tx=0).stop 2023-09-30T01:54:45.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:45.897+0000 7f16d0540700 10 monclient: shutdown 2023-09-30T01:54:45.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:45.897+0000 7f16d0540700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:45.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:45.897+0000 7f16d0540700 1 -- 172.21.15.7:0/1900425101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16c8096e20 msgr2=0x7f16c812da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:45.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:45.897+0000 7f16d0540700 1 --2- 172.21.15.7:0/1900425101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16c8096e20 0x7f16c812da80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f16c80a2b60 tx=0x7f16bc016000 comp rx=0 tx=0).stop 2023-09-30T01:54:45.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:45.897+0000 7f16d0540700 1 -- 172.21.15.7:0/1900425101 shutdown_connections 2023-09-30T01:54:45.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:45.897+0000 7f16d0540700 1 --2- 172.21.15.7:0/1900425101 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f16b40357d0 0x7f16b4037c80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:45.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:45.897+0000 7f16d0540700 1 --2- 172.21.15.7:0/1900425101 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16c812ef00 0x7f16c812dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:45.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:45.897+0000 7f16d0540700 1 --2- 172.21.15.7:0/1900425101 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f16c8096a50 0x7f16c812d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:45.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:45.897+0000 7f16d0540700 1 --2- 172.21.15.7:0/1900425101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f16c8096e20 0x7f16c812da80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:45.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:45.897+0000 7f16d0540700 1 -- 172.21.15.7:0/1900425101 >> 172.21.15.7:0/1900425101 conn(0x7f16c8070cf0 msgr2=0x7f16c8131410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:45.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:45.897+0000 7f16d0540700 1 -- 172.21.15.7:0/1900425101 shutdown_connections 2023-09-30T01:54:45.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:45.897+0000 7f16d0540700 1 -- 172.21.15.7:0/1900425101 wait complete. 2023-09-30T01:54:45.898 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'rbd' created 2023-09-30T01:54:45.911 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-30T01:54:46.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.043+0000 7fc96afd0700 10 monclient: get_monmap_and_config 2023-09-30T01:54:46.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.043+0000 7fc96afd0700 10 monclient: build_initial_monmap 2023-09-30T01:54:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.043+0000 7fc96afd0700 10 monclient: monmap: 2023-09-30T01:54:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:46.044567+0000 2023-09-30T01:54:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:46.044567+0000 2023-09-30T01:54:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:46.044 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:46.045 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:46.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.044+0000 7fc96afd0700 1 Processor -- start 2023-09-30T01:54:46.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.044+0000 7fc96afd0700 1 -- start start 2023-09-30T01:54:46.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.044+0000 7fc96afd0700 10 monclient: init 2023-09-30T01:54:46.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.044+0000 7fc96afd0700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:46.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.044+0000 7fc96afd0700 10 monclient: _add_conns ranks=[3,2,4,1,0] 2023-09-30T01:54:46.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.044+0000 7fc96afd0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc9640a2900 0x7fc964096a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:46.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.044+0000 7fc96afd0700 10 monclient(hunting): picked mon.noname-a con 0x7fc9640a2900 addr v2:172.21.15.7:3300/0 2023-09-30T01:54:46.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.044+0000 7fc96afd0700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc964096f90 0x7fc9640973b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:46.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.044+0000 7fc96afd0700 10 monclient(hunting): picked mon.noname-b con 0x7fc964096f90 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:46.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.044+0000 7fc96afd0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc9641321b0 0x7fc964132600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:46.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.044+0000 7fc96afd0700 10 monclient(hunting): picked mon.noname-c con 0x7fc9641321b0 addr v2:172.21.15.50:3300/0 2023-09-30T01:54:46.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.044+0000 7fc96afd0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:46.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.044+0000 7fc96afd0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9640978f0 con 0x7fc9640a2900 2023-09-30T01:54:46.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.044+0000 7fc96afd0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:46.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.044+0000 7fc96afd0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc964098ea0 con 0x7fc9641321b0 2023-09-30T01:54:46.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.044+0000 7fc96afd0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:46.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.044+0000 7fc96afd0700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9640991c0 con 0x7fc964096f90 2023-09-30T01:54:46.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.044+0000 7fc96afd0700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:46.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.044+0000 7fc96afd0700 10 monclient(hunting): authenticate will time out at 1175.109871s 2023-09-30T01:54:46.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.044+0000 7fc963fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc964096f90 0x7fc9640973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:46.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.044+0000 7fc968d6c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc9640a2900 0x7fc964096a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:46.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc968d6c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc9640a2900 0x7fc964096a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54624/0 (socket says 172.21.15.7:54624) 2023-09-30T01:54:46.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc963fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc964096f90 0x7fc9640973b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59252/0 (socket says 172.21.15.7:59252) 2023-09-30T01:54:46.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc968d6c700 1 -- 172.21.15.7:0/2794049199 learned_addr learned my addr 172.21.15.7:0/2794049199 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:46.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc968d6c700 10 monclient(hunting): get_auth_request con 0x7fc9640a2900 auth_method 0 2023-09-30T01:54:46.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc968d6c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:46.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc968d6c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:46.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc968d6c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:46.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc963fff700 10 monclient(hunting): get_auth_request con 0x7fc964096f90 auth_method 0 2023-09-30T01:54:46.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc963fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:46.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc963fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:46.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc963fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:46.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc96956d700 1 --2- 172.21.15.7:0/2794049199 >> v2:172.21.15.50:3300/0 conn(0x7fc9641321b0 0x7fc964132600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:46.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc96956d700 10 monclient(hunting): get_auth_request con 0x7fc9641321b0 auth_method 0 2023-09-30T01:54:46.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc96956d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:46.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc96956d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:46.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc96956d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:46.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc968d6c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:46.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc968d6c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:46.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc968d6c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:46.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc963fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:46.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc963fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:46.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc963fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:46.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc96956d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:46.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc96956d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:46.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc96956d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:46.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc968d6c700 10 monclient(hunting): handle_auth_done global_id 4203 payload 274 2023-09-30T01:54:46.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc968d6c700 1 -- 172.21.15.7:0/2794049199 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc964096f90 msgr2=0x7fc9640973b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:46.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc968d6c700 1 --2- 172.21.15.7:0/2794049199 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc964096f90 0x7fc9640973b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:46.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc968d6c700 1 -- 172.21.15.7:0/2794049199 >> v2:172.21.15.50:3300/0 conn(0x7fc9641321b0 msgr2=0x7fc964132600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:46.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc968d6c700 1 --2- 172.21.15.7:0/2794049199 >> v2:172.21.15.50:3300/0 conn(0x7fc9641321b0 0x7fc964132600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:46.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc968d6c700 10 monclient: _finish_hunting 0 2023-09-30T01:54:46.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc968d6c700 1 monclient: found mon.noname-a 2023-09-30T01:54:46.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc968d6c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:46.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc968d6c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:46.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc968d6c700 1 -- 172.21.15.7:0/2794049199 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9640a16d0 con 0x7fc9640a2900 2023-09-30T01:54:46.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.045+0000 7fc963fff700 1 --2- 172.21.15.7:0/2794049199 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc964096f90 0x7fc9640973b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:46.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.047+0000 7fc968d6c700 1 -- 172.21.15.7:0/2794049199 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9640a2900 msgr2=0x7fc964096a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:46.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.047+0000 7fc968d6c700 1 --2- 172.21.15.7:0/2794049199 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9640a2900 0x7fc964096a50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc9640a31b0 tx=0x7fc958009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7dd2a69d2a54a6e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:46.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.047+0000 7fc9637fe700 1 -- 172.21.15.7:0/2794049199 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95800e070 con 0x7fc9640a2900 2023-09-30T01:54:46.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.047+0000 7fc9637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:46.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.047+0000 7fc9637fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:46.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.047+0000 7fc9637fe700 10 monclient: dump: 2023-09-30T01:54:46.054 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:46.054 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:46.054 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:46.054 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:46.054 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:46.055 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:46.055 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:46.055 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:46.055 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:46.055 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:46.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.047+0000 7fc9637fe700 10 monclient: _finish_auth 0 2023-09-30T01:54:46.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.047+0000 7fc9637fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:46.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.047+0000 7fc9637fe700 1 -- 172.21.15.7:0/2794049199 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95800bb80 con 0x7fc9640a2900 2023-09-30T01:54:46.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.047+0000 7fc96afd0700 5 monclient: authenticate success, global_id 4203 2023-09-30T01:54:46.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.047+0000 7fc96afd0700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:46.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.047+0000 7fc9637fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:46.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.047+0000 7fc9637fe700 1 -- 172.21.15.7:0/2794049199 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9580092e0 con 0x7fc9640a2900 2023-09-30T01:54:46.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.047+0000 7fc96afd0700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:46.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.047+0000 7fc96afd0700 4 set_mon_vals no callback set 2023-09-30T01:54:46.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.047+0000 7fc9637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:46.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.047+0000 7fc9637fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:46.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.047+0000 7fc9637fe700 10 monclient: dump: 2023-09-30T01:54:46.056 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:46.056 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:46.056 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:46.056 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:46.056 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:46.056 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:46.056 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:46.056 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:46.057 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:46.057 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:46.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.048+0000 7fc96afd0700 10 monclient: shutdown 2023-09-30T01:54:46.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.048+0000 7fc96afd0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:46.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.048+0000 7fc96afd0700 1 -- 172.21.15.7:0/2794049199 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9640a2900 msgr2=0x7fc964096a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:46.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.048+0000 7fc96afd0700 1 --2- 172.21.15.7:0/2794049199 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9640a2900 0x7fc964096a50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc9640a31b0 tx=0x7fc958009840 comp rx=0 tx=0).stop 2023-09-30T01:54:46.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.048+0000 7fc96afd0700 1 -- 172.21.15.7:0/2794049199 shutdown_connections 2023-09-30T01:54:46.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.048+0000 7fc96afd0700 1 --2- 172.21.15.7:0/2794049199 >> v2:172.21.15.50:3300/0 conn(0x7fc9641321b0 0x7fc964132600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:46.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.048+0000 7fc96afd0700 1 --2- 172.21.15.7:0/2794049199 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9640a2900 0x7fc964096a50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:46.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.048+0000 7fc96afd0700 1 --2- 172.21.15.7:0/2794049199 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc964096f90 0x7fc9640973b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:46.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.048+0000 7fc96afd0700 1 -- 172.21.15.7:0/2794049199 >> 172.21.15.7:0/2794049199 conn(0x7fc9640a5cd0 msgr2=0x7fc9640a21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:46.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.048+0000 7fc96afd0700 1 -- 172.21.15.7:0/2794049199 shutdown_connections 2023-09-30T01:54:46.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.048+0000 7fc96afd0700 1 -- 172.21.15.7:0/2794049199 wait complete. 2023-09-30T01:54:46.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.048+0000 7fc96afd0700 10 monclient: build_initial_monmap 2023-09-30T01:54:46.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.048+0000 7fc96afd0700 10 monclient: monmap: 2023-09-30T01:54:46.058 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:46.058 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:46.058 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:46.058 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:46.058 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:46.058 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:46.058 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:46.058 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:46.058 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:46.058 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:46.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.049+0000 7fc96afd0700 1 Processor -- start 2023-09-30T01:54:46.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.049+0000 7fc96afd0700 1 -- start start 2023-09-30T01:54:46.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.049+0000 7fc96afd0700 10 monclient: init 2023-09-30T01:54:46.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.049+0000 7fc96afd0700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:46.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.049+0000 7fc96afd0700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:54:46.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.049+0000 7fc96afd0700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc964096f90 0x7fc964141f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:46.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.049+0000 7fc96afd0700 10 monclient(hunting): picked mon.noname-c con 0x7fc964096f90 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:46.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.049+0000 7fc96afd0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9640a2900 0x7fc96409d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:46.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.049+0000 7fc96afd0700 10 monclient(hunting): picked mon.noname-b con 0x7fc9640a2900 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:46.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.049+0000 7fc96afd0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9641321b0 0x7fc96409dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:46.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.049+0000 7fc96afd0700 10 monclient(hunting): picked mon.noname-a con 0x7fc9641321b0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:46.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.049+0000 7fc96afd0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:46.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.049+0000 7fc96afd0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc964130480 con 0x7fc9641321b0 2023-09-30T01:54:46.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.049+0000 7fc96afd0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:46.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.049+0000 7fc96afd0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96411c620 con 0x7fc9640a2900 2023-09-30T01:54:46.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.049+0000 7fc96afd0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:46.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.049+0000 7fc96afd0700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc96411bd60 con 0x7fc964096f90 2023-09-30T01:54:46.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.049+0000 7fc96afd0700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:46.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.049+0000 7fc96afd0700 10 monclient(hunting): authenticate will time out at 1175.114940s 2023-09-30T01:54:46.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc96956d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9641321b0 0x7fc96409dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:46.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc968d6c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc964096f90 0x7fc964141f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:46.065 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc96956d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9641321b0 0x7fc96409dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54638/0 (socket says 172.21.15.7:54638) 2023-09-30T01:54:46.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc96956d700 1 -- 172.21.15.7:0/4139298287 learned_addr learned my addr 172.21.15.7:0/4139298287 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:46.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc96956d700 10 monclient(hunting): get_auth_request con 0x7fc9641321b0 auth_method 0 2023-09-30T01:54:46.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc96956d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:46.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc96956d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:46.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc96956d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:46.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc968d6c700 10 monclient(hunting): get_auth_request con 0x7fc964096f90 auth_method 0 2023-09-30T01:54:46.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc968d6c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:46.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc968d6c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:46.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc968d6c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:46.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc963fff700 1 --2- 172.21.15.7:0/4139298287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9640a2900 0x7fc96409d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:46.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc963fff700 10 monclient(hunting): get_auth_request con 0x7fc9640a2900 auth_method 0 2023-09-30T01:54:46.066 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc963fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:46.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc963fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:46.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc963fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:46.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc96956d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:46.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc96956d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:46.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc96956d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:46.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc968d6c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:46.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc968d6c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:46.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc968d6c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:46.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc963fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:46.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc963fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:46.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc963fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:46.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc968d6c700 10 monclient(hunting): handle_auth_done global_id 4202 payload 824 2023-09-30T01:54:46.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc968d6c700 1 -- 172.21.15.7:0/4139298287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9640a2900 msgr2=0x7fc96409d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:46.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc968d6c700 1 --2- 172.21.15.7:0/4139298287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9640a2900 0x7fc96409d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:46.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc968d6c700 1 -- 172.21.15.7:0/4139298287 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9641321b0 msgr2=0x7fc96409dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:46.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc968d6c700 1 --2- 172.21.15.7:0/4139298287 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9641321b0 0x7fc96409dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:46.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc968d6c700 10 monclient: _finish_hunting 0 2023-09-30T01:54:46.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc968d6c700 1 monclient: found mon.noname-c 2023-09-30T01:54:46.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc968d6c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:46.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc968d6c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:54:46.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc968d6c700 1 -- 172.21.15.7:0/4139298287 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc950009420 con 0x7fc964096f90 2023-09-30T01:54:46.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.050+0000 7fc96956d700 1 --2- 172.21.15.7:0/4139298287 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9641321b0 0x7fc96409dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:46.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc968d6c700 1 -- 172.21.15.7:0/4139298287 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc964096f90 msgr2=0x7fc964141f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:46.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc968d6c700 1 --2- 172.21.15.7:0/4139298287 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc964096f90 0x7fc964141f80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc958015000 tx=0x7fc958018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:46.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc960ff9700 1 -- 172.21.15.7:0/4139298287 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc958007820 con 0x7fc964096f90 2023-09-30T01:54:46.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc960ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:46.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc960ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:46.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc960ff9700 10 monclient: dump: 2023-09-30T01:54:46.069 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:46.069 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:46.069 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:46.069 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:46.069 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:46.069 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:46.069 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:46.069 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:46.069 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:46.069 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:46.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc960ff9700 10 monclient: _finish_auth 0 2023-09-30T01:54:46.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc960ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:46.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc960ff9700 1 -- 172.21.15.7:0/4139298287 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9580079c0 con 0x7fc964096f90 2023-09-30T01:54:46.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc96afd0700 5 monclient: authenticate success, global_id 4202 2023-09-30T01:54:46.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc96afd0700 10 monclient: _renew_subs 2023-09-30T01:54:46.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc96afd0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:46.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc96afd0700 1 -- 172.21.15.7:0/4139298287 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc958009ae0 con 0x7fc964096f90 2023-09-30T01:54:46.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc960ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:46.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc960ff9700 1 -- 172.21.15.7:0/4139298287 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc958007d80 con 0x7fc964096f90 2023-09-30T01:54:46.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc960ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:46.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc960ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:46.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc960ff9700 10 monclient: dump: 2023-09-30T01:54:46.071 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:46.071 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:46.071 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:46.071 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:46.071 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:46.071 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:46.071 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:46.071 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:46.071 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:46.071 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:46.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc962ffd700 4 set_mon_vals no callback set 2023-09-30T01:54:46.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc96afd0700 10 monclient: _renew_subs 2023-09-30T01:54:46.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc96afd0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:46.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.052+0000 7fc96afd0700 1 -- 172.21.15.7:0/4139298287 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9640a0070 con 0x7fc964096f90 2023-09-30T01:54:46.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.053+0000 7fc960ff9700 1 -- 172.21.15.7:0/4139298287 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 4) v1 ==== 42485+0+0 (secure 0 0 0) 0x7fc95801d420 con 0x7fc964096f90 2023-09-30T01:54:46.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.053+0000 7fc96afd0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:46.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.053+0000 7fc96afd0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:46.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.053+0000 7fc96afd0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:46.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.053+0000 7fc960ff9700 1 --2- 172.21.15.7:0/4139298287 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc948035750 0x7fc948037c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:46.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.053+0000 7fc96afd0700 1 -- 172.21.15.7:0/4139298287 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc96411c170 con 0x7fc964096f90 2023-09-30T01:54:46.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.053+0000 7fc960ff9700 1 -- 172.21.15.7:0/4139298287 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4331+0+0 (secure 0 0 0) 0x7fc958049590 con 0x7fc964096f90 2023-09-30T01:54:46.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.053+0000 7fc963fff700 1 --2- 172.21.15.7:0/4139298287 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc948035750 0x7fc948037c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:46.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.053+0000 7fc963fff700 10 monclient: get_auth_request con 0x7fc948035750 auth_method 0 2023-09-30T01:54:46.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.056+0000 7fc963fff700 1 --2- 172.21.15.7:0/4139298287 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc948035750 0x7fc948037c00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc96409f030 tx=0x7fc9500091a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:46.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.058+0000 7fc960ff9700 1 -- 172.21.15.7:0/4139298287 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc958021070 con 0x7fc964096f90 2023-09-30T01:54:46.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.058+0000 7fc960ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:46.073 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.058+0000 7fc960ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:46.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.259+0000 7fc96afd0700 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-09-30T01:54:46.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.259+0000 7fc96afd0700 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-09-30T01:54:46.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.259+0000 7fc96afd0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:46.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.259+0000 7fc96afd0700 1 -- 172.21.15.7:0/4139298287 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc964064c10 con 0x7fc964096f90 2023-09-30T01:54:46.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.893+0000 7fc960ff9700 1 -- 172.21.15.7:0/4139298287 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v14) v1 ==== 173+0+0 (secure 0 0 0) 0x7fc95802b030 con 0x7fc964096f90 2023-09-30T01:54:46.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.893+0000 7fc960ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-09-30T01:54:46.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.893+0000 7fc960ff9700 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2023-09-30T01:54:46.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.896+0000 7fc96afd0700 1 -- 172.21.15.7:0/4139298287 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc948035750 msgr2=0x7fc948037c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:46.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.896+0000 7fc96afd0700 1 --2- 172.21.15.7:0/4139298287 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc948035750 0x7fc948037c00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc96409f030 tx=0x7fc9500091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:46.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.896+0000 7fc96afd0700 10 monclient: shutdown 2023-09-30T01:54:46.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.896+0000 7fc96afd0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:46.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.896+0000 7fc96afd0700 1 -- 172.21.15.7:0/4139298287 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc964096f90 msgr2=0x7fc964141f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:46.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.896+0000 7fc96afd0700 1 --2- 172.21.15.7:0/4139298287 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc964096f90 0x7fc964141f80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc958015000 tx=0x7fc958018000 comp rx=0 tx=0).stop 2023-09-30T01:54:46.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.896+0000 7fc96afd0700 1 -- 172.21.15.7:0/4139298287 shutdown_connections 2023-09-30T01:54:46.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.896+0000 7fc96afd0700 1 --2- 172.21.15.7:0/4139298287 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc948035750 0x7fc948037c00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:46.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.896+0000 7fc96afd0700 1 --2- 172.21.15.7:0/4139298287 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc9641321b0 0x7fc96409dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:46.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.896+0000 7fc96afd0700 1 --2- 172.21.15.7:0/4139298287 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc964096f90 0x7fc964141f80 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc958015000 tx=0x7fc958018000 comp rx=0 tx=0).stop 2023-09-30T01:54:46.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.896+0000 7fc96afd0700 1 --2- 172.21.15.7:0/4139298287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc9640a2900 0x7fc96409d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:46.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.896+0000 7fc96afd0700 1 -- 172.21.15.7:0/4139298287 >> 172.21.15.7:0/4139298287 conn(0x7fc9640a5cd0 msgr2=0x7fc9640a4490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:46.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.898+0000 7fc96afd0700 1 -- 172.21.15.7:0/4139298287 shutdown_connections 2023-09-30T01:54:46.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:46.898+0000 7fc96afd0700 1 -- 172.21.15.7:0/4139298287 wait complete. 2023-09-30T01:54:46.899 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-30T01:54:46.912 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-30T01:54:46.912 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-30T01:54:46.912 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-30T01:54:46.912 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-30T01:54:46.913 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-30T01:54:47.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.036+0000 7eff4785e700 10 monclient: get_monmap_and_config 2023-09-30T01:54:47.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.036+0000 7eff4785e700 10 monclient: build_initial_monmap 2023-09-30T01:54:47.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.036+0000 7eff4785e700 10 monclient: monmap: 2023-09-30T01:54:47.037 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:47.037 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:47.037 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:47.037196+0000 2023-09-30T01:54:47.037 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:47.037196+0000 2023-09-30T01:54:47.037 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:47.037 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:47.037 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:47.037 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:47.037 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:47.037 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:47.037 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:47.037 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:47.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.036+0000 7eff4785e700 1 Processor -- start 2023-09-30T01:54:47.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.036+0000 7eff4785e700 1 -- start start 2023-09-30T01:54:47.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.036+0000 7eff4785e700 10 monclient: init 2023-09-30T01:54:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff4785e700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff4785e700 10 monclient: _add_conns ranks=[0,3,1,2,4] 2023-09-30T01:54:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff4785e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff400a2170 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff4785e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7eff4006cde0 0x7eff4006c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff4785e700 10 monclient(hunting): picked mon.noname-a con 0x7eff4006cde0 addr v2:172.21.15.7:3300/0 2023-09-30T01:54:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff4785e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7eff4013bb50 addr v1:172.21.15.50:6789/0 2023-09-30T01:54:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff4785e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff400a2540 con 0x7eff400a2170 2023-09-30T01:54:47.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff4785e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff4006cba0 con 0x7eff4013bb50 2023-09-30T01:54:47.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff4785e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:47.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff4785e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff4006f8b0 con 0x7eff4006cde0 2023-09-30T01:54:47.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff4785e700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:47.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff4785e700 10 monclient(hunting): authenticate will time out at 1176.102386s 2023-09-30T01:54:47.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff44df9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7eff4006cde0 0x7eff4006c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:47.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff455fa700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7eff400a2170 0x7eff4006b950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58060/0 (socket says 172.21.15.7:58060) 2023-09-30T01:54:47.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff455fa700 1 -- 172.21.15.7:0/2443608247 learned_addr learned my addr 172.21.15.7:0/2443608247 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:54:47.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff44df9700 10 monclient(hunting): get_auth_request con 0x7eff4006cde0 auth_method 0 2023-09-30T01:54:47.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff44df9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:47.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff44df9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:47.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff44df9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff44df9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff44df9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff44df9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff44df9700 10 monclient(hunting): handle_auth_done global_id 4212 payload 274 2023-09-30T01:54:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff44df9700 1 -- 172.21.15.7:0/2443608247 >> v1:172.21.15.50:6789/0 conn(0x7eff4013bb50 legacy=0x7eff4013df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff44df9700 1 -- 172.21.15.7:0/2443608247 >> v1:172.21.15.7:6789/0 conn(0x7eff400a2170 legacy=0x7eff4006b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff44df9700 10 monclient: _finish_hunting 0 2023-09-30T01:54:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff44df9700 1 monclient: found mon.noname-a 2023-09-30T01:54:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff44df9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff44df9700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.037+0000 7eff44df9700 1 -- 172.21.15.7:0/2443608247 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff4006e710 con 0x7eff4006cde0 2023-09-30T01:54:47.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.040+0000 7eff44df9700 1 -- 172.21.15.7:0/2443608247 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff4006cde0 msgr2=0x7eff4006c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.040+0000 7eff44df9700 1 --2- 172.21.15.7:0/2443608247 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff4006cde0 0x7eff4006c060 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7eff340098e0 tx=0x7eff340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f987f6bdb4945f04 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.040+0000 7eff3ffff700 1 -- 172.21.15.7:0/2443608247 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3400b9f0 con 0x7eff4006cde0 2023-09-30T01:54:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.040+0000 7eff3ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:47.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.040+0000 7eff3ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.040+0000 7eff3ffff700 10 monclient: dump: 2023-09-30T01:54:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:47.045 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:47.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.040+0000 7eff3ffff700 10 monclient: _finish_auth 0 2023-09-30T01:54:47.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.040+0000 7eff3ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:47.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.040+0000 7eff3ffff700 1 -- 172.21.15.7:0/2443608247 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff3400bb50 con 0x7eff4006cde0 2023-09-30T01:54:47.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.040+0000 7eff4785e700 5 monclient: authenticate success, global_id 4212 2023-09-30T01:54:47.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.040+0000 7eff4785e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:47.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.040+0000 7eff3ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:47.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.040+0000 7eff4785e700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:47.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.040+0000 7eff3ffff700 1 -- 172.21.15.7:0/2443608247 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff34011420 con 0x7eff4006cde0 2023-09-30T01:54:47.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.040+0000 7eff4785e700 4 set_mon_vals no callback set 2023-09-30T01:54:47.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.040+0000 7eff3ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:47.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.040+0000 7eff3ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:47.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.040+0000 7eff3ffff700 10 monclient: dump: 2023-09-30T01:54:47.046 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:47.047 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:47.047 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.047 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.047 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:47.047 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:47.047 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:47.047 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:47.047 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:47.047 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:47.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.041+0000 7eff4785e700 10 monclient: shutdown 2023-09-30T01:54:47.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.041+0000 7eff4785e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:47.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.041+0000 7eff4785e700 1 -- 172.21.15.7:0/2443608247 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff4006cde0 msgr2=0x7eff4006c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:47.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.041+0000 7eff4785e700 1 --2- 172.21.15.7:0/2443608247 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff4006cde0 0x7eff4006c060 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7eff340098e0 tx=0x7eff340091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.041+0000 7eff4785e700 1 -- 172.21.15.7:0/2443608247 shutdown_connections 2023-09-30T01:54:47.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.041+0000 7eff4785e700 1 --2- 172.21.15.7:0/2443608247 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff4006cde0 0x7eff4006c060 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.041+0000 7eff4785e700 1 -- 172.21.15.7:0/2443608247 >> 172.21.15.7:0/2443608247 conn(0x7eff40070cf0 msgr2=0x7eff400710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:47.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.041+0000 7eff4785e700 1 -- 172.21.15.7:0/2443608247 shutdown_connections 2023-09-30T01:54:47.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.041+0000 7eff4785e700 1 -- 172.21.15.7:0/2443608247 wait complete. 2023-09-30T01:54:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.042+0000 7eff4785e700 10 monclient: build_initial_monmap 2023-09-30T01:54:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.042+0000 7eff4785e700 10 monclient: monmap: 2023-09-30T01:54:47.052 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:47.053 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.042+0000 7eff4785e700 1 Processor -- start 2023-09-30T01:54:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.042+0000 7eff4785e700 1 -- start start 2023-09-30T01:54:47.053 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.042+0000 7eff4785e700 10 monclient: init 2023-09-30T01:54:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff4785e700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff4785e700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:54:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff4785e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff4006cde0 0x7eff400a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff4785e700 10 monclient(hunting): picked mon.noname-a con 0x7eff4006cde0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff4785e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff400a2170 0x7eff40146030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff4785e700 10 monclient(hunting): picked mon.noname-b con 0x7eff400a2170 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff4785e700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7eff4013bb50 0x7eff40146570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff4785e700 10 monclient(hunting): picked mon.noname-c con 0x7eff4013bb50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff4785e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff4785e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4009e1c0 con 0x7eff4006cde0 2023-09-30T01:54:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff4785e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:47.054 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff4785e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4009dee0 con 0x7eff400a2170 2023-09-30T01:54:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff4785e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff4785e700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff4009e4a0 con 0x7eff4013bb50 2023-09-30T01:54:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff4785e700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff4785e700 10 monclient(hunting): authenticate will time out at 1176.108131s 2023-09-30T01:54:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff455fa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff4006cde0 0x7eff400a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff45dfb700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7eff4013bb50 0x7eff40146570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff455fa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff4006cde0 0x7eff400a0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54642/0 (socket says 172.21.15.7:54642) 2023-09-30T01:54:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff455fa700 1 -- 172.21.15.7:0/3180888847 learned_addr learned my addr 172.21.15.7:0/3180888847 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff455fa700 10 monclient(hunting): get_auth_request con 0x7eff4006cde0 auth_method 0 2023-09-30T01:54:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff455fa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff455fa700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:47.055 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff455fa700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff45dfb700 10 monclient(hunting): get_auth_request con 0x7eff4013bb50 auth_method 0 2023-09-30T01:54:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff45dfb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff45dfb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff45dfb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff44df9700 1 --2- 172.21.15.7:0/3180888847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff400a2170 0x7eff40146030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff455fa700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff455fa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff455fa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff45dfb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff45dfb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff45dfb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:47.056 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff44df9700 10 monclient(hunting): get_auth_request con 0x7eff400a2170 auth_method 0 2023-09-30T01:54:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff44df9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff44df9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff44df9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff44df9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff44df9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff44df9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff455fa700 10 monclient(hunting): handle_auth_done global_id 4215 payload 824 2023-09-30T01:54:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff455fa700 1 -- 172.21.15.7:0/3180888847 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7eff4013bb50 msgr2=0x7eff40146570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff455fa700 1 --2- 172.21.15.7:0/3180888847 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7eff4013bb50 0x7eff40146570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff455fa700 1 -- 172.21.15.7:0/3180888847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff400a2170 msgr2=0x7eff40146030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff455fa700 1 --2- 172.21.15.7:0/3180888847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff400a2170 0x7eff40146030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.057 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff455fa700 10 monclient: _finish_hunting 0 2023-09-30T01:54:47.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff455fa700 1 monclient: found mon.noname-a 2023-09-30T01:54:47.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff455fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:47.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff455fa700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:47.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff455fa700 1 -- 172.21.15.7:0/3180888847 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff34009420 con 0x7eff4006cde0 2023-09-30T01:54:47.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.043+0000 7eff45dfb700 1 --2- 172.21.15.7:0/3180888847 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7eff4013bb50 0x7eff40146570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:47.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.045+0000 7eff455fa700 1 -- 172.21.15.7:0/3180888847 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff4006cde0 msgr2=0x7eff400a0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:47.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.045+0000 7eff455fa700 1 --2- 172.21.15.7:0/3180888847 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff4006cde0 0x7eff400a0cc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7eff3000cca0 tx=0x7eff30005480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:47.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.046+0000 7eff3d7fa700 1 -- 172.21.15.7:0/3180888847 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3000ab20 con 0x7eff4006cde0 2023-09-30T01:54:47.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.046+0000 7eff3d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:47.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.046+0000 7eff3d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:47.058 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.046+0000 7eff3d7fa700 10 monclient: dump: 2023-09-30T01:54:47.058 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:47.059 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:47.059 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.059 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.059 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:47.059 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:47.059 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:47.059 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:47.059 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:47.059 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:47.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.046+0000 7eff3d7fa700 10 monclient: _finish_auth 0 2023-09-30T01:54:47.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.046+0000 7eff3d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:47.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.046+0000 7eff4785e700 5 monclient: authenticate success, global_id 4215 2023-09-30T01:54:47.059 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.046+0000 7eff4785e700 10 monclient: _renew_subs 2023-09-30T01:54:47.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.046+0000 7eff3d7fa700 1 -- 172.21.15.7:0/3180888847 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff3000aca0 con 0x7eff4006cde0 2023-09-30T01:54:47.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.046+0000 7eff4785e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:47.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.046+0000 7eff4785e700 1 -- 172.21.15.7:0/3180888847 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff4009d080 con 0x7eff4006cde0 2023-09-30T01:54:47.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.046+0000 7eff3d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:47.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.046+0000 7eff3d7fa700 1 -- 172.21.15.7:0/3180888847 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff30007340 con 0x7eff4006cde0 2023-09-30T01:54:47.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.046+0000 7eff3d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:47.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.046+0000 7eff3d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:47.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.046+0000 7eff3d7fa700 10 monclient: dump: 2023-09-30T01:54:47.060 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:47.060 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:47.060 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.060 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.061 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:47.061 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:47.061 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:47.061 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:47.061 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:47.061 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:47.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.046+0000 7eff4785e700 10 monclient: _renew_subs 2023-09-30T01:54:47.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.046+0000 7eff4785e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:47.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.046+0000 7eff4785e700 1 -- 172.21.15.7:0/3180888847 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff400a13c0 con 0x7eff4006cde0 2023-09-30T01:54:47.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.046+0000 7eff3f7fe700 4 set_mon_vals no callback set 2023-09-30T01:54:47.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.046+0000 7eff3d7fa700 1 -- 172.21.15.7:0/3180888847 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 4) v1 ==== 42485+0+0 (secure 0 0 0) 0x7eff30007730 con 0x7eff4006cde0 2023-09-30T01:54:47.061 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.047+0000 7eff3d7fa700 1 --2- 172.21.15.7:0/3180888847 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7eff2c035750 0x7eff2c037c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:47.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.047+0000 7eff3d7fa700 1 -- 172.21.15.7:0/3180888847 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4342+0+0 (secure 0 0 0) 0x7eff300480a0 con 0x7eff4006cde0 2023-09-30T01:54:47.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.047+0000 7eff4785e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:47.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.047+0000 7eff4785e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:47.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.047+0000 7eff4785e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:47.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.047+0000 7eff4785e700 1 -- 172.21.15.7:0/3180888847 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff40064c10 con 0x7eff4006cde0 2023-09-30T01:54:47.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.052+0000 7eff3d7fa700 1 -- 172.21.15.7:0/3180888847 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7eff30010d10 con 0x7eff4006cde0 2023-09-30T01:54:47.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.052+0000 7eff3d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:47.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.052+0000 7eff3d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:47.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.055+0000 7eff44df9700 1 --2- 172.21.15.7:0/3180888847 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7eff2c035750 0x7eff2c037c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:47.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.055+0000 7eff44df9700 10 monclient: get_auth_request con 0x7eff2c035750 auth_method 0 2023-09-30T01:54:47.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.058+0000 7eff44df9700 1 --2- 172.21.15.7:0/3180888847 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7eff2c035750 0x7eff2c037c00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7eff4009e8c0 tx=0x7eff34016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:47.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.306+0000 7eff4785e700 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2023-09-30T01:54:47.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.306+0000 7eff4785e700 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2023-09-30T01:54:47.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.306+0000 7eff4785e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:47.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.306+0000 7eff4785e700 1 -- 172.21.15.7:0/3180888847 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7eff4006f560 con 0x7eff4006cde0 2023-09-30T01:54:47.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.308+0000 7eff3d7fa700 1 -- 172.21.15.7:0/3180888847 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) v1 ==== 74+0+82343 (secure 0 0 0) 0x7eff30010300 con 0x7eff4006cde0 2023-09-30T01:54:47.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.308+0000 7eff3d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2023-09-30T01:54:47.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.308+0000 7eff3d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:47.309 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:47.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.310+0000 7eff4785e700 1 -- 172.21.15.7:0/3180888847 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7eff2c035750 msgr2=0x7eff2c037c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:47.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.310+0000 7eff4785e700 1 --2- 172.21.15.7:0/3180888847 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7eff2c035750 0x7eff2c037c00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7eff4009e8c0 tx=0x7eff34016000 comp rx=0 tx=0).stop 2023-09-30T01:54:47.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.310+0000 7eff4785e700 10 monclient: shutdown 2023-09-30T01:54:47.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.310+0000 7eff4785e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:47.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.310+0000 7eff4785e700 1 -- 172.21.15.7:0/3180888847 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff4006cde0 msgr2=0x7eff400a0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:47.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.310+0000 7eff4785e700 1 --2- 172.21.15.7:0/3180888847 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff4006cde0 0x7eff400a0cc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7eff3000cca0 tx=0x7eff30005480 comp rx=0 tx=0).stop 2023-09-30T01:54:47.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.310+0000 7eff4785e700 1 -- 172.21.15.7:0/3180888847 shutdown_connections 2023-09-30T01:54:47.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.310+0000 7eff4785e700 1 --2- 172.21.15.7:0/3180888847 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7eff2c035750 0x7eff2c037c00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.310+0000 7eff4785e700 1 --2- 172.21.15.7:0/3180888847 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff4006cde0 0x7eff400a0cc0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.310+0000 7eff4785e700 1 --2- 172.21.15.7:0/3180888847 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7eff4013bb50 0x7eff40146570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.310+0000 7eff4785e700 1 --2- 172.21.15.7:0/3180888847 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff400a2170 0x7eff40146030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.310+0000 7eff4785e700 1 -- 172.21.15.7:0/3180888847 >> 172.21.15.7:0/3180888847 conn(0x7eff40070cf0 msgr2=0x7eff400a3670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:47.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.310+0000 7eff4785e700 1 -- 172.21.15.7:0/3180888847 shutdown_connections 2023-09-30T01:54:47.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.310+0000 7eff4785e700 1 -- 172.21.15.7:0/3180888847 wait complete. 2023-09-30T01:54:47.322 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":4,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":86200},{"type":"v1","addr":"172.21.15.50:6825","nonce":86200}]},"active_addr":"172.21.15.50:6825/86200","active_change":"2023-09-30T01:54:40.258394+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":1962899897}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":1677535374}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":192651537}]}]} 2023-09-30T01:54:47.324 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-30T01:54:47.324 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-30T01:54:47.325 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:54:47.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.442+0000 7f972ea27700 10 monclient: get_monmap_and_config 2023-09-30T01:54:47.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.442+0000 7f972ea27700 10 monclient: build_initial_monmap 2023-09-30T01:54:47.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.442+0000 7f972ea27700 10 monclient: monmap: 2023-09-30T01:54:47.443 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:47.443 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:47.443 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:47.443585+0000 2023-09-30T01:54:47.443 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:47.443585+0000 2023-09-30T01:54:47.443 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:47.443 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:47.443 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:47.443 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:47.443 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:47.444 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:47.444 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:47.444 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:47.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f972ea27700 1 Processor -- start 2023-09-30T01:54:47.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f972ea27700 1 -- start start 2023-09-30T01:54:47.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f972ea27700 10 monclient: init 2023-09-30T01:54:47.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f972ea27700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:47.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f972ea27700 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-09-30T01:54:47.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f972ea27700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9728096a50 0x7f97281327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:47.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f972ea27700 10 monclient(hunting): picked mon.noname-b con 0x7f9728096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:47.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f972ea27700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f97280a2390 0x7f9728096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:47.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f972ea27700 10 monclient(hunting): picked mon.noname-a con 0x7f97280a2390 addr v2:172.21.15.7:3300/0 2023-09-30T01:54:47.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f972ea27700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9728097360 addr v1:172.21.15.50:6789/0 2023-09-30T01:54:47.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f972ea27700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97280980e0 con 0x7f9728097360 2023-09-30T01:54:47.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f972ea27700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:47.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f972ea27700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97280991a0 con 0x7f97280a2390 2023-09-30T01:54:47.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f972ea27700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:47.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f972ea27700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9728099480 con 0x7f9728096a50 2023-09-30T01:54:47.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f972ea27700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:47.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f972ea27700 10 monclient(hunting): authenticate will time out at 1176.508750s 2023-09-30T01:54:47.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f97277fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f97280a2390 0x7f9728096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:47.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f9727fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9728096a50 0x7f97281327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:47.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f97277fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f97280a2390 0x7f9728096e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54650/0 (socket says 172.21.15.7:54650) 2023-09-30T01:54:47.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f97277fe700 1 -- 172.21.15.7:0/638486718 learned_addr learned my addr 172.21.15.7:0/638486718 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:47.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f97277fe700 10 monclient(hunting): get_auth_request con 0x7f97280a2390 auth_method 0 2023-09-30T01:54:47.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f9727fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9728096a50 0x7f97281327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59276/0 (socket says 172.21.15.7:59276) 2023-09-30T01:54:47.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f97277fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:47.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f97277fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:47.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f97277fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:47.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f9727fff700 10 monclient(hunting): get_auth_request con 0x7f9728096a50 auth_method 0 2023-09-30T01:54:47.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f9727fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:47.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f9727fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:47.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.443+0000 7f9727fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:47.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.444+0000 7f97277fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:47.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.444+0000 7f97277fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:47.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.444+0000 7f97277fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:47.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.444+0000 7f9727fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:47.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.444+0000 7f9727fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:47.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.444+0000 7f9727fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:47.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.444+0000 7f97277fe700 10 monclient(hunting): handle_auth_done global_id 4218 payload 274 2023-09-30T01:54:47.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.444+0000 7f97277fe700 1 -- 172.21.15.7:0/638486718 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9728096a50 msgr2=0x7f97281327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:47.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.444+0000 7f97277fe700 1 --2- 172.21.15.7:0/638486718 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9728096a50 0x7f97281327a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.444+0000 7f97277fe700 1 -- 172.21.15.7:0/638486718 >> v1:172.21.15.50:6789/0 conn(0x7f9728097360 legacy=0x7f972812ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:47.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.444+0000 7f97277fe700 10 monclient: _finish_hunting 0 2023-09-30T01:54:47.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.444+0000 7f97277fe700 1 monclient: found mon.noname-a 2023-09-30T01:54:47.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.444+0000 7f97277fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:47.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.444+0000 7f97277fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:47.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.444+0000 7f97277fe700 1 -- 172.21.15.7:0/638486718 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9728097bf0 con 0x7f97280a2390 2023-09-30T01:54:47.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.444+0000 7f9727fff700 1 --2- 172.21.15.7:0/638486718 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9728096a50 0x7f97281327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:47.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.446+0000 7f97277fe700 1 -- 172.21.15.7:0/638486718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97280a2390 msgr2=0x7f9728096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:47.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.446+0000 7f97277fe700 1 --2- 172.21.15.7:0/638486718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97280a2390 0x7f9728096e20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f971c0099b0 tx=0x7f971c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89a0b1aa835ff380 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:47.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.446+0000 7f9726ffd700 1 -- 172.21.15.7:0/638486718 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971c009ea0 con 0x7f97280a2390 2023-09-30T01:54:47.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.446+0000 7f9726ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:47.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.446+0000 7f9726ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:47.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.446+0000 7f9726ffd700 10 monclient: dump: 2023-09-30T01:54:47.456 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:47.456 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:47.456 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.456 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.456 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:47.456 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:47.456 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:47.456 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:47.456 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:47.457 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:47.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.446+0000 7f9726ffd700 10 monclient: _finish_auth 0 2023-09-30T01:54:47.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.446+0000 7f9726ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:47.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.446+0000 7f9726ffd700 1 -- 172.21.15.7:0/638486718 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f971c00ba30 con 0x7f97280a2390 2023-09-30T01:54:47.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.446+0000 7f972ea27700 5 monclient: authenticate success, global_id 4218 2023-09-30T01:54:47.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.446+0000 7f972ea27700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:47.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.446+0000 7f9726ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:47.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.446+0000 7f9726ffd700 1 -- 172.21.15.7:0/638486718 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971c00bdf0 con 0x7f97280a2390 2023-09-30T01:54:47.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.446+0000 7f9726ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:47.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.446+0000 7f9726ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:47.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.446+0000 7f9726ffd700 10 monclient: dump: 2023-09-30T01:54:47.458 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:47.458 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:47.458 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.458 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.458 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:47.458 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:47.458 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:47.458 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:47.458 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:47.458 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:47.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.446+0000 7f972ea27700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:47.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.446+0000 7f972ea27700 4 set_mon_vals no callback set 2023-09-30T01:54:47.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.447+0000 7f972ea27700 10 monclient: shutdown 2023-09-30T01:54:47.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.447+0000 7f972ea27700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:47.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.447+0000 7f972ea27700 1 -- 172.21.15.7:0/638486718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97280a2390 msgr2=0x7f9728096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:47.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.447+0000 7f972ea27700 1 --2- 172.21.15.7:0/638486718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97280a2390 0x7f9728096e20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f971c0099b0 tx=0x7f971c009250 comp rx=0 tx=0).stop 2023-09-30T01:54:47.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.447+0000 7f972ea27700 1 -- 172.21.15.7:0/638486718 shutdown_connections 2023-09-30T01:54:47.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.447+0000 7f972ea27700 1 --2- 172.21.15.7:0/638486718 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9728096a50 0x7f97281327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.447+0000 7f972ea27700 1 --2- 172.21.15.7:0/638486718 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97280a2390 0x7f9728096e20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.459 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.447+0000 7f972ea27700 1 -- 172.21.15.7:0/638486718 >> 172.21.15.7:0/638486718 conn(0x7f9728070cf0 msgr2=0x7f97280710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:47.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.447+0000 7f972ea27700 1 -- 172.21.15.7:0/638486718 shutdown_connections 2023-09-30T01:54:47.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.447+0000 7f972ea27700 1 -- 172.21.15.7:0/638486718 wait complete. 2023-09-30T01:54:47.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.447+0000 7f972ea27700 10 monclient: build_initial_monmap 2023-09-30T01:54:47.460 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.447+0000 7f972ea27700 10 monclient: monmap: 2023-09-30T01:54:47.460 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:47.460 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:47.460 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:47.460 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:47.460 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:47.460 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:47.460 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:47.460 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:47.461 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:47.461 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:47.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972ea27700 1 Processor -- start 2023-09-30T01:54:47.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972ea27700 1 -- start start 2023-09-30T01:54:47.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972ea27700 10 monclient: init 2023-09-30T01:54:47.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972ea27700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:47.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972ea27700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:54:47.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972ea27700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9728096a50 0x7f972812dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:47.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972ea27700 10 monclient(hunting): picked mon.noname-c con 0x7f9728096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:47.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972ea27700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9728097360 0x7f972809db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:47.461 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972ea27700 10 monclient(hunting): picked mon.noname-b con 0x7f9728097360 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:47.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972ea27700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97280a2390 0x7f972809e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:47.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972ea27700 10 monclient(hunting): picked mon.noname-a con 0x7f97280a2390 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:47.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972ea27700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:47.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972ea27700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97281334e0 con 0x7f97280a2390 2023-09-30T01:54:47.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972ea27700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:47.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972ea27700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972812f030 con 0x7f9728097360 2023-09-30T01:54:47.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972ea27700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:47.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972ea27700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97280a1530 con 0x7f9728096a50 2023-09-30T01:54:47.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972ea27700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:47.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972ea27700 10 monclient(hunting): authenticate will time out at 1176.513804s 2023-09-30T01:54:47.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f9727fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9728096a50 0x7f972812dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:47.462 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f9727fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9728096a50 0x7f972812dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59286/0 (socket says 172.21.15.7:59286) 2023-09-30T01:54:47.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f9727fff700 1 -- 172.21.15.7:0/3111210180 learned_addr learned my addr 172.21.15.7:0/3111210180 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:47.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f9727fff700 10 monclient(hunting): get_auth_request con 0x7f9728096a50 auth_method 0 2023-09-30T01:54:47.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f9727fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:47.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f9727fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:47.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f9727fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:47.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972cfc4700 1 --2- 172.21.15.7:0/3111210180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97280a2390 0x7f972809e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:47.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972cfc4700 10 monclient(hunting): get_auth_request con 0x7f97280a2390 auth_method 0 2023-09-30T01:54:47.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972cfc4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:47.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972cfc4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:47.463 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f972cfc4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:47.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f9727fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:47.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f9727fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:47.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.448+0000 7f9727fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:47.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.449+0000 7f972cfc4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:47.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.449+0000 7f972cfc4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:47.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.449+0000 7f972cfc4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:47.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.449+0000 7f97277fe700 1 --2- 172.21.15.7:0/3111210180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9728097360 0x7f972809db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:47.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.449+0000 7f97277fe700 10 monclient(hunting): get_auth_request con 0x7f9728097360 auth_method 0 2023-09-30T01:54:47.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.449+0000 7f97277fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:47.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.449+0000 7f97277fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:47.464 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.449+0000 7f97277fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:47.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.449+0000 7f972cfc4700 10 monclient(hunting): handle_auth_done global_id 4221 payload 824 2023-09-30T01:54:47.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.449+0000 7f972cfc4700 1 -- 172.21.15.7:0/3111210180 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9728096a50 msgr2=0x7f972812dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:47.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.449+0000 7f972cfc4700 1 --2- 172.21.15.7:0/3111210180 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9728096a50 0x7f972812dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.449+0000 7f972cfc4700 1 -- 172.21.15.7:0/3111210180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9728097360 msgr2=0x7f972809db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:47.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.449+0000 7f972cfc4700 1 --2- 172.21.15.7:0/3111210180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9728097360 0x7f972809db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.449+0000 7f972cfc4700 10 monclient: _finish_hunting 0 2023-09-30T01:54:47.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.449+0000 7f972cfc4700 1 monclient: found mon.noname-a 2023-09-30T01:54:47.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.449+0000 7f972cfc4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:47.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.449+0000 7f972cfc4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:47.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.449+0000 7f972cfc4700 1 -- 172.21.15.7:0/3111210180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9718009420 con 0x7f97280a2390 2023-09-30T01:54:47.465 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.449+0000 7f9727fff700 1 --2- 172.21.15.7:0/3111210180 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9728096a50 0x7f972812dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:47.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f972cfc4700 1 -- 172.21.15.7:0/3111210180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97280a2390 msgr2=0x7f972809e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:47.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f972cfc4700 1 --2- 172.21.15.7:0/3111210180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97280a2390 0x7f972809e0c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f972000c810 tx=0x7f9720002620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:47.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f970ffff700 1 -- 172.21.15.7:0/3111210180 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9720012840 con 0x7f97280a2390 2023-09-30T01:54:47.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f970ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:47.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f970ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:47.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f970ffff700 10 monclient: dump: 2023-09-30T01:54:47.468 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:47.468 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:47.468 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.469 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.469 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:47.469 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:47.469 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:47.469 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:47.469 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:47.469 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:47.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f970ffff700 10 monclient: _finish_auth 0 2023-09-30T01:54:47.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f970ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:47.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f970ffff700 1 -- 172.21.15.7:0/3111210180 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97200129a0 con 0x7f97280a2390 2023-09-30T01:54:47.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f972ea27700 5 monclient: authenticate success, global_id 4221 2023-09-30T01:54:47.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f972ea27700 10 monclient: _renew_subs 2023-09-30T01:54:47.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f972ea27700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:47.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f972ea27700 1 -- 172.21.15.7:0/3111210180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971c0094f0 con 0x7f97280a2390 2023-09-30T01:54:47.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f970ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:47.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f970ffff700 1 -- 172.21.15.7:0/3111210180 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9720012d60 con 0x7f97280a2390 2023-09-30T01:54:47.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f970ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:47.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f970ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:47.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f970ffff700 10 monclient: dump: 2023-09-30T01:54:47.470 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:47.470 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:47.470 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.471 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.471 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:47.471 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:47.471 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:47.471 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:47.471 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:47.471 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:47.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f972ea27700 10 monclient: _renew_subs 2023-09-30T01:54:47.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f972ea27700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:47.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f972ea27700 1 -- 172.21.15.7:0/3111210180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97281428a0 con 0x7f97280a2390 2023-09-30T01:54:47.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.451+0000 7f97267fc700 4 set_mon_vals no callback set 2023-09-30T01:54:47.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.452+0000 7f970ffff700 1 -- 172.21.15.7:0/3111210180 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f9720011590 con 0x7f97280a2390 2023-09-30T01:54:47.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.452+0000 7f970ffff700 1 --2- 172.21.15.7:0/3111210180 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f97140357e0 0x7f9714037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:47.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.452+0000 7f970ffff700 1 -- 172.21.15.7:0/3111210180 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f9720047fe0 con 0x7f97280a2390 2023-09-30T01:54:47.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.452+0000 7f972ea27700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:47.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.452+0000 7f972ea27700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:47.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.452+0000 7f972ea27700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:47.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.452+0000 7f972ea27700 1 -- 172.21.15.7:0/3111210180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97280a4810 con 0x7f97280a2390 2023-09-30T01:54:47.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.452+0000 7f9727fff700 1 --2- 172.21.15.7:0/3111210180 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f97140357e0 0x7f9714037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:47.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.452+0000 7f9727fff700 10 monclient: get_auth_request con 0x7f97140357e0 auth_method 0 2023-09-30T01:54:47.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.455+0000 7f9727fff700 1 --2- 172.21.15.7:0/3111210180 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f97140357e0 0x7f9714037c90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f971800f680 tx=0x7f97180091a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:47.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.457+0000 7f970ffff700 1 -- 172.21.15.7:0/3111210180 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9720016070 con 0x7f97280a2390 2023-09-30T01:54:47.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.457+0000 7f970ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:47.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.457+0000 7f970ffff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:47.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.649+0000 7f972ea27700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:54:47.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.649+0000 7f972ea27700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:54:47.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.649+0000 7f972ea27700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:47.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.649+0000 7f972ea27700 1 -- 172.21.15.7:0/3111210180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97280a16a0 con 0x7f97280a2390 2023-09-30T01:54:47.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.650+0000 7f970ffff700 1 -- 172.21.15.7:0/3111210180 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+11249 (secure 0 0 0) 0x7f97200104c0 con 0x7f97280a2390 2023-09-30T01:54:47.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.650+0000 7f970ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:54:47.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.650+0000 7f970ffff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:47.651 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:47.651 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":14,"fsid":"2ea1044d-d984-49b9-baad-7f36bd454a19","created":"2023-09-30T01:54:34.356209+0000","modified":"2023-09-30T01:54:46.890579+0000","last_up_change":"2023-09-30T01:54:43.359934+0000","last_in_change":"2023-09-30T01:54:36.628886+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T01:54:44.376777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T01:54:45.091042+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"e44e9247-cb86-4a16-ac1f-f021cdf5108f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":86601},{"type":"v1","addr":"172.21.15.7:6801","nonce":86601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":86601},{"type":"v1","addr":"172.21.15.7:6803","nonce":86601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":86601},{"type":"v1","addr":"172.21.15.7:6807","nonce":86601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":86601},{"type":"v1","addr":"172.21.15.7:6805","nonce":86601}]},"public_addr":"172.21.15.7:6801/86601","cluster_addr":"172.21.15.7:6803/86601","heartbeat_back_addr":"172.21.15.7:6807/86601","heartbeat_front_addr":"172.21.15.7:6805/86601","state":["exists","up"]},{"osd":1,"uuid":"e30730fd-1e49-44bf-bdc9-087e23727df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":86600},{"type":"v1","addr":"172.21.15.7:6817","nonce":86600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":86600},{"type":"v1","addr":"172.21.15.7:6819","nonce":86600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":86600},{"type":"v1","addr":"172.21.15.7:6823","nonce":86600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":86600},{"type":"v1","addr":"172.21.15.7:6821","nonce":86600}]},"public_addr":"172.21.15.7:6817/86600","cluster_addr":"172.21.15.7:6819/86600","heartbeat_back_addr":"172.21.15.7:6823/86600","heartbeat_front_addr":"172.21.15.7:6821/86600","state":["exists","up"]},{"osd":2,"uuid":"b39c552e-f5cc-4e4f-90a3-5aa5f5127a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":86602},{"type":"v1","addr":"172.21.15.7:6809","nonce":86602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":86602},{"type":"v1","addr":"172.21.15.7:6811","nonce":86602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":86602},{"type":"v1","addr":"172.21.15.7:6815","nonce":86602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":86602},{"type":"v1","addr":"172.21.15.7:6813","nonce":86602}]},"public_addr":"172.21.15.7:6809/86602","cluster_addr":"172.21.15.7:6811/86602","heartbeat_back_addr":"172.21.15.7:6815/86602","heartbeat_front_addr":"172.21.15.7:6813/86602","state":["exists","up"]},{"osd":3,"uuid":"15cd89ae-049a-4e1a-8c6b-7d086ca6e1df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":86633},{"type":"v1","addr":"172.21.15.50:6807","nonce":86633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":86633},{"type":"v1","addr":"172.21.15.50:6812","nonce":86633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6819","nonce":86633},{"type":"v1","addr":"172.21.15.50:6821","nonce":86633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6815","nonce":86633},{"type":"v1","addr":"172.21.15.50:6818","nonce":86633}]},"public_addr":"172.21.15.50:6807/86633","cluster_addr":"172.21.15.50:6812/86633","heartbeat_back_addr":"172.21.15.50:6821/86633","heartbeat_front_addr":"172.21.15.50:6818/86633","state":["exists","up"]},{"osd":4,"uuid":"de6773bd-d286-4ab9-b91f-8a90ee3073d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":86632},{"type":"v1","addr":"172.21.15.50:6806","nonce":86632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":86632},{"type":"v1","addr":"172.21.15.50:6814","nonce":86632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":86632},{"type":"v1","addr":"172.21.15.50:6823","nonce":86632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":86632},{"type":"v1","addr":"172.21.15.50:6820","nonce":86632}]},"public_addr":"172.21.15.50:6806/86632","cluster_addr":"172.21.15.50:6814/86632","heartbeat_back_addr":"172.21.15.50:6823/86632","heartbeat_front_addr":"172.21.15.50:6820/86632","state":["exists","up"]},{"osd":5,"uuid":"e6a847ab-276e-4d54-9a8f-6df5b8ab54cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":86634},{"type":"v1","addr":"172.21.15.50:6801","nonce":86634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":86634},{"type":"v1","addr":"172.21.15.50:6805","nonce":86634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":86634},{"type":"v1","addr":"172.21.15.50:6816","nonce":86634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":86634},{"type":"v1","addr":"172.21.15.50:6811","nonce":86634}]},"public_addr":"172.21.15.50:6801/86634","cluster_addr":"172.21.15.50:6805/86634","heartbeat_back_addr":"172.21.15.50:6816/86634","heartbeat_front_addr":"172.21.15.50:6811/86634","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.819964+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.795270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.734518+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.507975+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.783023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.746894+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:54:47.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.652+0000 7f972ea27700 1 -- 172.21.15.7:0/3111210180 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f97140357e0 msgr2=0x7f9714037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:47.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.652+0000 7f972ea27700 1 --2- 172.21.15.7:0/3111210180 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f97140357e0 0x7f9714037c90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f971800f680 tx=0x7f97180091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.652+0000 7f972ea27700 10 monclient: shutdown 2023-09-30T01:54:47.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.652+0000 7f972ea27700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:47.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.652+0000 7f972ea27700 1 -- 172.21.15.7:0/3111210180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97280a2390 msgr2=0x7f972809e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:47.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.652+0000 7f972ea27700 1 --2- 172.21.15.7:0/3111210180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97280a2390 0x7f972809e0c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f972000c810 tx=0x7f9720002620 comp rx=0 tx=0).stop 2023-09-30T01:54:47.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.652+0000 7f972ea27700 1 -- 172.21.15.7:0/3111210180 shutdown_connections 2023-09-30T01:54:47.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.652+0000 7f972ea27700 1 --2- 172.21.15.7:0/3111210180 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f97140357e0 0x7f9714037c90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.652+0000 7f972ea27700 1 --2- 172.21.15.7:0/3111210180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97280a2390 0x7f972809e0c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.652+0000 7f972ea27700 1 --2- 172.21.15.7:0/3111210180 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9728096a50 0x7f972812dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.652+0000 7f972ea27700 1 --2- 172.21.15.7:0/3111210180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9728097360 0x7f972809db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.652+0000 7f972ea27700 1 -- 172.21.15.7:0/3111210180 >> 172.21.15.7:0/3111210180 conn(0x7f9728070cf0 msgr2=0x7f9728131a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:47.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.652+0000 7f972ea27700 1 -- 172.21.15.7:0/3111210180 shutdown_connections 2023-09-30T01:54:47.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.652+0000 7f972ea27700 1 -- 172.21.15.7:0/3111210180 wait complete. 2023-09-30T01:54:47.664 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-30T01:54:47.664 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:54:47.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.782+0000 7fa40c1a7700 10 monclient: get_monmap_and_config 2023-09-30T01:54:47.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.782+0000 7fa40c1a7700 10 monclient: build_initial_monmap 2023-09-30T01:54:47.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.782+0000 7fa40c1a7700 10 monclient: monmap: 2023-09-30T01:54:47.782 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:47.783 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:47.783 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:47.783430+0000 2023-09-30T01:54:47.783 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:47.783430+0000 2023-09-30T01:54:47.783 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:47.783 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:47.783 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:47.783 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:47.783 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:47.784 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:47.784 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:47.784 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:47.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.782+0000 7fa40c1a7700 1 Processor -- start 2023-09-30T01:54:47.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40c1a7700 1 -- start start 2023-09-30T01:54:47.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40c1a7700 10 monclient: init 2023-09-30T01:54:47.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40c1a7700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:47.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40c1a7700 10 monclient: _add_conns ranks=[4,1,2,0,3] 2023-09-30T01:54:47.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40c1a7700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa4040a29e0 0x7fa404096a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:47.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40c1a7700 10 monclient(hunting): picked mon.noname-c con 0x7fa4040a29e0 addr v2:172.21.15.50:3300/0 2023-09-30T01:54:47.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40c1a7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa404096f90 addr v1:172.21.15.50:6789/0 2023-09-30T01:54:47.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40c1a7700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa404097390 0x7fa404132930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:47.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40c1a7700 10 monclient(hunting): picked mon.noname-b con 0x7fa404097390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:47.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40c1a7700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa404069cc0 con 0x7fa404096f90 2023-09-30T01:54:47.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40c1a7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:47.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40c1a7700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa404098ea0 con 0x7fa4040a29e0 2023-09-30T01:54:47.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40c1a7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:47.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40c1a7700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4040a2db0 con 0x7fa404097390 2023-09-30T01:54:47.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40c1a7700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:47.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40c1a7700 10 monclient(hunting): authenticate will time out at 1176.848613s 2023-09-30T01:54:47.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40a744700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa404097390 0x7fa404132930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:47.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40a744700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa404097390 0x7fa404132930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59302/0 (socket says 172.21.15.7:59302) 2023-09-30T01:54:47.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40a744700 1 -- 172.21.15.7:0/3285603767 learned_addr learned my addr 172.21.15.7:0/3285603767 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:47.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40a744700 10 monclient(hunting): get_auth_request con 0x7fa404097390 auth_method 0 2023-09-30T01:54:47.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40a744700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:47.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40a744700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:47.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40a744700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:47.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40a744700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:47.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40a744700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:47.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa40a744700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:47.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa409f43700 1 --2- 172.21.15.7:0/3285603767 >> v2:172.21.15.50:3300/0 conn(0x7fa4040a29e0 0x7fa404096a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:47.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa409f43700 10 monclient(hunting): get_auth_request con 0x7fa4040a29e0 auth_method 0 2023-09-30T01:54:47.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa409f43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:47.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa409f43700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:47.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.783+0000 7fa409f43700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:47.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.784+0000 7fa409f43700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:47.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.784+0000 7fa409f43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:47.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.784+0000 7fa409f43700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:47.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.784+0000 7fa40a744700 10 monclient(hunting): handle_auth_done global_id 4217 payload 274 2023-09-30T01:54:47.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.784+0000 7fa40a744700 1 -- 172.21.15.7:0/3285603767 >> v2:172.21.15.50:3300/0 conn(0x7fa4040a29e0 msgr2=0x7fa404096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:47.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.784+0000 7fa40a744700 1 --2- 172.21.15.7:0/3285603767 >> v2:172.21.15.50:3300/0 conn(0x7fa4040a29e0 0x7fa404096a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.784+0000 7fa40a744700 1 -- 172.21.15.7:0/3285603767 >> v1:172.21.15.50:6789/0 conn(0x7fa404096f90 legacy=0x7fa404132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:47.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.784+0000 7fa40a744700 10 monclient: _finish_hunting 0 2023-09-30T01:54:47.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.784+0000 7fa40a744700 1 monclient: found mon.noname-b 2023-09-30T01:54:47.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.784+0000 7fa40a744700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:47.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.784+0000 7fa40a744700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:54:47.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.784+0000 7fa40a744700 1 -- 172.21.15.7:0/3285603767 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa404098270 con 0x7fa404097390 2023-09-30T01:54:47.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.786+0000 7fa40a744700 1 -- 172.21.15.7:0/3285603767 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa404097390 msgr2=0x7fa404132930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:47.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.786+0000 7fa40a744700 1 --2- 172.21.15.7:0/3285603767 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa404097390 0x7fa404132930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc009f20 tx=0x7fa3fc009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b519810c6671600 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:47.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.786+0000 7fa408f41700 1 -- 172.21.15.7:0/3285603767 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc00e070 con 0x7fa404097390 2023-09-30T01:54:47.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.786+0000 7fa408f41700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:47.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.786+0000 7fa408f41700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:54:47.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.786+0000 7fa408f41700 10 monclient: dump: 2023-09-30T01:54:47.799 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:47.799 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:47.799 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.799 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.800 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:47.800 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:47.800 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:47.800 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:47.800 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:47.800 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:47.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.786+0000 7fa408f41700 10 monclient: _finish_auth 0 2023-09-30T01:54:47.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.786+0000 7fa408f41700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:47.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.786+0000 7fa40c1a7700 5 monclient: authenticate success, global_id 4217 2023-09-30T01:54:47.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.786+0000 7fa40c1a7700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:47.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.786+0000 7fa408f41700 1 -- 172.21.15.7:0/3285603767 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3fc00bba0 con 0x7fa404097390 2023-09-30T01:54:47.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.786+0000 7fa408f41700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:47.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.786+0000 7fa40c1a7700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:47.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.786+0000 7fa40c1a7700 4 set_mon_vals no callback set 2023-09-30T01:54:47.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.786+0000 7fa408f41700 1 -- 172.21.15.7:0/3285603767 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc0092e0 con 0x7fa404097390 2023-09-30T01:54:47.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.786+0000 7fa408f41700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:47.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.786+0000 7fa408f41700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:47.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.786+0000 7fa408f41700 10 monclient: dump: 2023-09-30T01:54:47.802 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:47.802 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:47.802 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.802 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.802 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:47.802 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:47.802 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:47.802 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:47.803 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:47.803 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:47.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.787+0000 7fa40c1a7700 10 monclient: shutdown 2023-09-30T01:54:47.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.787+0000 7fa40c1a7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:47.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.787+0000 7fa40c1a7700 1 -- 172.21.15.7:0/3285603767 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa404097390 msgr2=0x7fa404132930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:47.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.787+0000 7fa40c1a7700 1 --2- 172.21.15.7:0/3285603767 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa404097390 0x7fa404132930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc009f20 tx=0x7fa3fc009840 comp rx=0 tx=0).stop 2023-09-30T01:54:47.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.787+0000 7fa40c1a7700 1 -- 172.21.15.7:0/3285603767 shutdown_connections 2023-09-30T01:54:47.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.787+0000 7fa40c1a7700 1 --2- 172.21.15.7:0/3285603767 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa404097390 0x7fa404132930 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.787+0000 7fa40c1a7700 1 --2- 172.21.15.7:0/3285603767 >> v2:172.21.15.50:3300/0 conn(0x7fa4040a29e0 0x7fa404096a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.787+0000 7fa40c1a7700 1 -- 172.21.15.7:0/3285603767 >> 172.21.15.7:0/3285603767 conn(0x7fa4040710d0 msgr2=0x7fa4040714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:47.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.787+0000 7fa40c1a7700 1 -- 172.21.15.7:0/3285603767 shutdown_connections 2023-09-30T01:54:47.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.787+0000 7fa40c1a7700 1 -- 172.21.15.7:0/3285603767 wait complete. 2023-09-30T01:54:47.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.787+0000 7fa40c1a7700 10 monclient: build_initial_monmap 2023-09-30T01:54:47.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.787+0000 7fa40c1a7700 10 monclient: monmap: 2023-09-30T01:54:47.804 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:47.804 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:47.805 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:47.805 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:47.805 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:47.805 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:47.805 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:47.805 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:47.805 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:47.805 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:47.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40c1a7700 1 Processor -- start 2023-09-30T01:54:47.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40c1a7700 1 -- start start 2023-09-30T01:54:47.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40c1a7700 10 monclient: init 2023-09-30T01:54:47.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40c1a7700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:47.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40c1a7700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:54:47.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40c1a7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa404096f90 0x7fa4040a0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:47.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40c1a7700 10 monclient(hunting): picked mon.noname-a con 0x7fa404096f90 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:47.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40c1a7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa404097390 0x7fa40409bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:47.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40c1a7700 10 monclient(hunting): picked mon.noname-b con 0x7fa404097390 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:47.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40c1a7700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa4040a29e0 0x7fa40409c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:47.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40c1a7700 10 monclient(hunting): picked mon.noname-c con 0x7fa4040a29e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:47.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40c1a7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:47.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40c1a7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4040a1220 con 0x7fa404096f90 2023-09-30T01:54:47.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40c1a7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:47.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40c1a7700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4041350c0 con 0x7fa404097390 2023-09-30T01:54:47.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40c1a7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:47.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40c1a7700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa40409f6b0 con 0x7fa4040a29e0 2023-09-30T01:54:47.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40c1a7700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:47.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40c1a7700 10 monclient(hunting): authenticate will time out at 1176.853674s 2023-09-30T01:54:47.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa409f43700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa404096f90 0x7fa4040a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:47.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40a744700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa4040a29e0 0x7fa40409c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:47.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa409f43700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa404096f90 0x7fa4040a0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54662/0 (socket says 172.21.15.7:54662) 2023-09-30T01:54:47.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa409f43700 1 -- 172.21.15.7:0/3034816725 learned_addr learned my addr 172.21.15.7:0/3034816725 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:47.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa409f43700 10 monclient(hunting): get_auth_request con 0x7fa404096f90 auth_method 0 2023-09-30T01:54:47.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa409f43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:47.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa409f43700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:47.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa409f43700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:47.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40a744700 10 monclient(hunting): get_auth_request con 0x7fa4040a29e0 auth_method 0 2023-09-30T01:54:47.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40a744700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:47.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40a744700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:47.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40a744700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:47.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa409f43700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:47.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa409f43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:47.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa409f43700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:47.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40a744700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:47.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40a744700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:47.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa40a744700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:47.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.788+0000 7fa409742700 1 --2- 172.21.15.7:0/3034816725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa404097390 0x7fa40409bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:47.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.789+0000 7fa409742700 10 monclient(hunting): get_auth_request con 0x7fa404097390 auth_method 0 2023-09-30T01:54:47.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.789+0000 7fa409742700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:47.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.789+0000 7fa409742700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:47.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.789+0000 7fa409742700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:47.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.789+0000 7fa40a744700 10 monclient(hunting): handle_auth_done global_id 4220 payload 824 2023-09-30T01:54:47.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.789+0000 7fa40a744700 1 -- 172.21.15.7:0/3034816725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa404097390 msgr2=0x7fa40409bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:47.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.789+0000 7fa40a744700 1 --2- 172.21.15.7:0/3034816725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa404097390 0x7fa40409bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.789+0000 7fa40a744700 1 -- 172.21.15.7:0/3034816725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa404096f90 msgr2=0x7fa4040a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:47.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.789+0000 7fa40a744700 1 --2- 172.21.15.7:0/3034816725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa404096f90 0x7fa4040a0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:47.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.789+0000 7fa40a744700 10 monclient: _finish_hunting 0 2023-09-30T01:54:47.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.789+0000 7fa40a744700 1 monclient: found mon.noname-c 2023-09-30T01:54:47.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.789+0000 7fa40a744700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:47.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.789+0000 7fa40a744700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:54:47.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.789+0000 7fa40a744700 1 -- 172.21.15.7:0/3034816725 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc009ae0 con 0x7fa4040a29e0 2023-09-30T01:54:47.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.789+0000 7fa409f43700 1 --2- 172.21.15.7:0/3034816725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa404096f90 0x7fa4040a0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:47.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa40a744700 1 -- 172.21.15.7:0/3034816725 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa4040a29e0 msgr2=0x7fa40409c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:47.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa40a744700 1 --2- 172.21.15.7:0/3034816725 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa4040a29e0 0x7fa40409c0f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa404098070 tx=0x7fa3fc017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:47.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa3f9ffb700 1 -- 172.21.15.7:0/3034816725 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc0071b0 con 0x7fa4040a29e0 2023-09-30T01:54:47.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa3f9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:47.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa3f9ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:47.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa3f9ffb700 10 monclient: dump: 2023-09-30T01:54:47.812 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:47.813 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:47.813 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.813 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.813 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:47.813 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:47.813 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:47.813 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:47.813 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:47.813 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:47.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa3f9ffb700 10 monclient: _finish_auth 0 2023-09-30T01:54:47.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa3f9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:47.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa3f9ffb700 1 -- 172.21.15.7:0/3034816725 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3fc012ae0 con 0x7fa4040a29e0 2023-09-30T01:54:47.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa40c1a7700 5 monclient: authenticate success, global_id 4220 2023-09-30T01:54:47.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa40c1a7700 10 monclient: _renew_subs 2023-09-30T01:54:47.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa40c1a7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:47.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa40c1a7700 1 -- 172.21.15.7:0/3034816725 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa40409e4a0 con 0x7fa4040a29e0 2023-09-30T01:54:47.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa3f9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:47.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa3f9ffb700 1 -- 172.21.15.7:0/3034816725 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc012e60 con 0x7fa4040a29e0 2023-09-30T01:54:47.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa3f9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:47.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa3f9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:47.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa3f9ffb700 10 monclient: dump: 2023-09-30T01:54:47.815 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:47.815 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:47.815 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.815 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:47.815 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:47.815 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:47.816 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:47.816 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:47.816 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:47.816 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:47.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa3fbfff700 4 set_mon_vals no callback set 2023-09-30T01:54:47.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa40c1a7700 10 monclient: _renew_subs 2023-09-30T01:54:47.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa40c1a7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:47.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.791+0000 7fa40c1a7700 1 -- 172.21.15.7:0/3034816725 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa40409d7e0 con 0x7fa4040a29e0 2023-09-30T01:54:47.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.792+0000 7fa3f9ffb700 1 -- 172.21.15.7:0/3034816725 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fa3fc01d630 con 0x7fa4040a29e0 2023-09-30T01:54:47.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.792+0000 7fa40c1a7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:47.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.792+0000 7fa40c1a7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:47.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.792+0000 7fa40c1a7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:47.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.792+0000 7fa40c1a7700 1 -- 172.21.15.7:0/3034816725 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa40409f830 con 0x7fa4040a29e0 2023-09-30T01:54:47.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.792+0000 7fa3f9ffb700 1 --2- 172.21.15.7:0/3034816725 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa3e8035830 0x7fa3e8037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:47.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.792+0000 7fa3f9ffb700 1 -- 172.21.15.7:0/3034816725 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fa3fc0495b0 con 0x7fa4040a29e0 2023-09-30T01:54:47.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.792+0000 7fa409f43700 1 --2- 172.21.15.7:0/3034816725 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa3e8035830 0x7fa3e8037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:47.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.793+0000 7fa409f43700 10 monclient: get_auth_request con 0x7fa3e8035830 auth_method 0 2023-09-30T01:54:47.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.795+0000 7fa409f43700 1 --2- 172.21.15.7:0/3034816725 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa3e8035830 0x7fa3e8037ce0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa400011520 tx=0x7fa400009120 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:47.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.797+0000 7fa3f9ffb700 1 -- 172.21.15.7:0/3034816725 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa40409f830 con 0x7fa4040a29e0 2023-09-30T01:54:47.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.797+0000 7fa3f9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:47.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.797+0000 7fa3f9ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:47.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.996+0000 7fa40c1a7700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:54:47.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.996+0000 7fa40c1a7700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:54:47.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.996+0000 7fa40c1a7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:47.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.996+0000 7fa40c1a7700 1 -- 172.21.15.7:0/3034816725 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa40409f830 con 0x7fa4040a29e0 2023-09-30T01:54:47.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.996+0000 7fa3f9ffb700 1 -- 172.21.15.7:0/3034816725 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+11236 (secure 0 0 0) 0x7fa40409f830 con 0x7fa4040a29e0 2023-09-30T01:54:47.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.996+0000 7fa3f9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:54:47.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.996+0000 7fa3f9ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:47.997 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:47.997 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":15,"fsid":"2ea1044d-d984-49b9-baad-7f36bd454a19","created":"2023-09-30T01:54:34.356209+0000","modified":"2023-09-30T01:54:47.895378+0000","last_up_change":"2023-09-30T01:54:43.359934+0000","last_in_change":"2023-09-30T01:54:36.628886+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T01:54:44.376777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T01:54:45.091042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"e44e9247-cb86-4a16-ac1f-f021cdf5108f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":86601},{"type":"v1","addr":"172.21.15.7:6801","nonce":86601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":86601},{"type":"v1","addr":"172.21.15.7:6803","nonce":86601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":86601},{"type":"v1","addr":"172.21.15.7:6807","nonce":86601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":86601},{"type":"v1","addr":"172.21.15.7:6805","nonce":86601}]},"public_addr":"172.21.15.7:6801/86601","cluster_addr":"172.21.15.7:6803/86601","heartbeat_back_addr":"172.21.15.7:6807/86601","heartbeat_front_addr":"172.21.15.7:6805/86601","state":["exists","up"]},{"osd":1,"uuid":"e30730fd-1e49-44bf-bdc9-087e23727df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":86600},{"type":"v1","addr":"172.21.15.7:6817","nonce":86600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":86600},{"type":"v1","addr":"172.21.15.7:6819","nonce":86600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":86600},{"type":"v1","addr":"172.21.15.7:6823","nonce":86600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":86600},{"type":"v1","addr":"172.21.15.7:6821","nonce":86600}]},"public_addr":"172.21.15.7:6817/86600","cluster_addr":"172.21.15.7:6819/86600","heartbeat_back_addr":"172.21.15.7:6823/86600","heartbeat_front_addr":"172.21.15.7:6821/86600","state":["exists","up"]},{"osd":2,"uuid":"b39c552e-f5cc-4e4f-90a3-5aa5f5127a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":86602},{"type":"v1","addr":"172.21.15.7:6809","nonce":86602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":86602},{"type":"v1","addr":"172.21.15.7:6811","nonce":86602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":86602},{"type":"v1","addr":"172.21.15.7:6815","nonce":86602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":86602},{"type":"v1","addr":"172.21.15.7:6813","nonce":86602}]},"public_addr":"172.21.15.7:6809/86602","cluster_addr":"172.21.15.7:6811/86602","heartbeat_back_addr":"172.21.15.7:6815/86602","heartbeat_front_addr":"172.21.15.7:6813/86602","state":["exists","up"]},{"osd":3,"uuid":"15cd89ae-049a-4e1a-8c6b-7d086ca6e1df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":86633},{"type":"v1","addr":"172.21.15.50:6807","nonce":86633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":86633},{"type":"v1","addr":"172.21.15.50:6812","nonce":86633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6819","nonce":86633},{"type":"v1","addr":"172.21.15.50:6821","nonce":86633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6815","nonce":86633},{"type":"v1","addr":"172.21.15.50:6818","nonce":86633}]},"public_addr":"172.21.15.50:6807/86633","cluster_addr":"172.21.15.50:6812/86633","heartbeat_back_addr":"172.21.15.50:6821/86633","heartbeat_front_addr":"172.21.15.50:6818/86633","state":["exists","up"]},{"osd":4,"uuid":"de6773bd-d286-4ab9-b91f-8a90ee3073d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":86632},{"type":"v1","addr":"172.21.15.50:6806","nonce":86632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":86632},{"type":"v1","addr":"172.21.15.50:6814","nonce":86632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":86632},{"type":"v1","addr":"172.21.15.50:6823","nonce":86632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":86632},{"type":"v1","addr":"172.21.15.50:6820","nonce":86632}]},"public_addr":"172.21.15.50:6806/86632","cluster_addr":"172.21.15.50:6814/86632","heartbeat_back_addr":"172.21.15.50:6823/86632","heartbeat_front_addr":"172.21.15.50:6820/86632","state":["exists","up"]},{"osd":5,"uuid":"e6a847ab-276e-4d54-9a8f-6df5b8ab54cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":86634},{"type":"v1","addr":"172.21.15.50:6801","nonce":86634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":86634},{"type":"v1","addr":"172.21.15.50:6805","nonce":86634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":86634},{"type":"v1","addr":"172.21.15.50:6816","nonce":86634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":86634},{"type":"v1","addr":"172.21.15.50:6811","nonce":86634}]},"public_addr":"172.21.15.50:6801/86634","cluster_addr":"172.21.15.50:6805/86634","heartbeat_back_addr":"172.21.15.50:6816/86634","heartbeat_front_addr":"172.21.15.50:6811/86634","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.819964+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.795270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.734518+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.507975+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.783023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.746894+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:54:47.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.999+0000 7fa40c1a7700 1 -- 172.21.15.7:0/3034816725 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa3e8035830 msgr2=0x7fa3e8037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:47.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.999+0000 7fa40c1a7700 1 --2- 172.21.15.7:0/3034816725 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa3e8035830 0x7fa3e8037ce0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa400011520 tx=0x7fa400009120 comp rx=0 tx=0).stop 2023-09-30T01:54:47.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.999+0000 7fa40c1a7700 10 monclient: shutdown 2023-09-30T01:54:48.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.999+0000 7fa40c1a7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.999+0000 7fa40c1a7700 1 -- 172.21.15.7:0/3034816725 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa4040a29e0 msgr2=0x7fa40409c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.999+0000 7fa40c1a7700 1 --2- 172.21.15.7:0/3034816725 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa4040a29e0 0x7fa40409c0f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa404098070 tx=0x7fa3fc017000 comp rx=0 tx=0).stop 2023-09-30T01:54:48.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.999+0000 7fa40c1a7700 1 -- 172.21.15.7:0/3034816725 shutdown_connections 2023-09-30T01:54:48.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.999+0000 7fa40c1a7700 1 --2- 172.21.15.7:0/3034816725 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa3e8035830 0x7fa3e8037ce0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.999+0000 7fa40c1a7700 1 --2- 172.21.15.7:0/3034816725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa404096f90 0x7fa4040a0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.999+0000 7fa40c1a7700 1 --2- 172.21.15.7:0/3034816725 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa4040a29e0 0x7fa40409c0f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.999+0000 7fa40c1a7700 1 --2- 172.21.15.7:0/3034816725 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa404097390 0x7fa40409bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.999+0000 7fa40c1a7700 1 -- 172.21.15.7:0/3034816725 >> 172.21.15.7:0/3034816725 conn(0x7fa4040710d0 msgr2=0x7fa4040a4060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.999+0000 7fa40c1a7700 1 -- 172.21.15.7:0/3034816725 shutdown_connections 2023-09-30T01:54:48.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:47.999+0000 7fa40c1a7700 1 -- 172.21.15.7:0/3034816725 wait complete. 2023-09-30T01:54:48.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-30T01:54:48.012 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-30T01:54:48.012 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-30T01:54:48.012 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-09-30T01:54:48.013 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-09-30T01:54:48.013 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-09-30T01:54:48.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.147+0000 7f92dd9a2700 10 monclient: get_monmap_and_config 2023-09-30T01:54:48.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.147+0000 7f92dd9a2700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.147+0000 7f92dd9a2700 10 monclient: monmap: 2023-09-30T01:54:48.148 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.148 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.148 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:48.148407+0000 2023-09-30T01:54:48.148 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:48.148407+0000 2023-09-30T01:54:48.148 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.148 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.148 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:48.148 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:48.148 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:48.148 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:48.148 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:48.148 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.147+0000 7fb4b936f700 10 monclient: get_monmap_and_config 2023-09-30T01:54:48.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.147+0000 7fb4b936f700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.147+0000 7fb4b936f700 10 monclient: monmap: 2023-09-30T01:54:48.149 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.149 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.149 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:48.148594+0000 2023-09-30T01:54:48.149 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:48.148594+0000 2023-09-30T01:54:48.150 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.150 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.150 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:48.150 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:48.150 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:48.150 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:48.150 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:48.150 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7f92dd9a2700 1 Processor -- start 2023-09-30T01:54:48.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7fb4b936f700 1 Processor -- start 2023-09-30T01:54:48.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7f92dd9a2700 1 -- start start 2023-09-30T01:54:48.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7f92dd9a2700 10 monclient: init 2023-09-30T01:54:48.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7f92dd9a2700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7f92dd9a2700 10 monclient: _add_conns ranks=[3,1,2,0,4] 2023-09-30T01:54:48.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7f92dd9a2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f92d8096a50 0x7f92d8132680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7f92dd9a2700 10 monclient(hunting): picked mon.noname-a con 0x7f92d8096a50 addr v2:172.21.15.7:3300/0 2023-09-30T01:54:48.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7f92dd9a2700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f92d8096e20 addr v1:172.21.15.50:6789/0 2023-09-30T01:54:48.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7f92dd9a2700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92d8137710 0x7f92d8139af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7f92dd9a2700 10 monclient(hunting): picked mon.noname-b con 0x7f92d8137710 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:48.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7f92dd9a2700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d8099440 con 0x7f92d8096e20 2023-09-30T01:54:48.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7f92dd9a2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7f92dd9a2700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92d8097b40 con 0x7f92d8096a50 2023-09-30T01:54:48.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7f92dd9a2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7f92dd9a2700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92d8099160 con 0x7f92d8137710 2023-09-30T01:54:48.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7f92dd9a2700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7f92dd9a2700 10 monclient(hunting): authenticate will time out at 1177.213760s 2023-09-30T01:54:48.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7fb4b936f700 1 -- start start 2023-09-30T01:54:48.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7fb4b936f700 10 monclient: init 2023-09-30T01:54:48.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7fb4b936f700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7fb4b936f700 10 monclient: _add_conns ranks=[0,4,2,3,1] 2023-09-30T01:54:48.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7fb4b936f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb4b409d250 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:48.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7fb4b936f700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb4b4132b70 0x7fb4b4134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7fb4b936f700 10 monclient(hunting): picked mon.noname-c con 0x7fb4b4132b70 addr v2:172.21.15.50:3300/0 2023-09-30T01:54:48.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7f92d6ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f92d8096a50 0x7f92d8132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.148+0000 7f92d6ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f92d8096a50 0x7f92d8132680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54664/0 (socket says 172.21.15.7:54664) 2023-09-30T01:54:48.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d6ffd700 1 -- 172.21.15.7:0/3373069497 learned_addr learned my addr 172.21.15.7:0/3373069497 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:48.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d6ffd700 10 monclient(hunting): get_auth_request con 0x7f92d8096a50 auth_method 0 2023-09-30T01:54:48.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d77fe700 1 --2- 172.21.15.7:0/3373069497 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92d8137710 0x7f92d8139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d77fe700 10 monclient(hunting): get_auth_request con 0x7f92d8137710 auth_method 0 2023-09-30T01:54:48.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7fb4b936f700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb4b4135490 0x7fb4b4137870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7fb4b936f700 10 monclient(hunting): picked mon.noname-b con 0x7fb4b4135490 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:48.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d6ffd700 10 monclient(hunting): handle_auth_done global_id 4227 payload 274 2023-09-30T01:54:48.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d6ffd700 1 -- 172.21.15.7:0/3373069497 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92d8137710 msgr2=0x7f92d8139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d6ffd700 1 --2- 172.21.15.7:0/3373069497 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92d8137710 0x7f92d8139af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d6ffd700 1 -- 172.21.15.7:0/3373069497 >> v1:172.21.15.50:6789/0 conn(0x7f92d8096e20 legacy=0x7f92d8097220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d6ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d6ffd700 1 monclient: found mon.noname-a 2023-09-30T01:54:48.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d6ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:48.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7f92d6ffd700 1 -- 172.21.15.7:0/3373069497 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d8099720 con 0x7f92d8096a50 2023-09-30T01:54:48.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7fb4b936f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b409ad20 con 0x7fb4b409d250 2023-09-30T01:54:48.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7fb4b936f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7fb4b936f700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4b409bde0 con 0x7fb4b4132b70 2023-09-30T01:54:48.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7fb4b936f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7fb4b936f700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b409bb00 con 0x7fb4b4135490 2023-09-30T01:54:48.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7fb4b936f700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7fb4b936f700 10 monclient(hunting): authenticate will time out at 1177.214838s 2023-09-30T01:54:48.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.149+0000 7fb4b37fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb4b4135490 0x7fb4b4137870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.150+0000 7fb4b37fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb4b4135490 0x7fb4b4137870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59330/0 (socket says 172.21.15.7:59330) 2023-09-30T01:54:48.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.150+0000 7fb4b37fe700 1 -- 172.21.15.7:0/4010692338 learned_addr learned my addr 172.21.15.7:0/4010692338 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:48.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.150+0000 7fb4b37fe700 10 monclient(hunting): get_auth_request con 0x7fb4b4135490 auth_method 0 2023-09-30T01:54:48.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.150+0000 7fb4b37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.150+0000 7fb4b37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.150+0000 7fb4b37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.150+0000 7fb4b37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.150+0000 7fb4b37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.150+0000 7fb4b37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.150+0000 7fb4b37fe700 10 monclient(hunting): handle_auth_done global_id 4226 payload 274 2023-09-30T01:54:48.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.150+0000 7fb4b37fe700 1 -- 172.21.15.7:0/4010692338 >> v2:172.21.15.50:3300/0 conn(0x7fb4b4132b70 msgr2=0x7fb4b4134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.150+0000 7fb4b37fe700 1 --2- 172.21.15.7:0/4010692338 >> v2:172.21.15.50:3300/0 conn(0x7fb4b4132b70 0x7fb4b4134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.150+0000 7fb4b37fe700 1 -- 172.21.15.7:0/4010692338 >> v1:172.21.15.7:6789/0 conn(0x7fb4b409d250 legacy=0x7fb4b409caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.150+0000 7fb4b37fe700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.150+0000 7fb4b37fe700 1 monclient: found mon.noname-b 2023-09-30T01:54:48.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.150+0000 7fb4b37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.150+0000 7fb4b37fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.150+0000 7fb4b37fe700 1 -- 172.21.15.7:0/4010692338 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b409aa70 con 0x7fb4b4135490 2023-09-30T01:54:48.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.151+0000 7f92d6ffd700 1 -- 172.21.15.7:0/3373069497 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92d8096a50 msgr2=0x7f92d8132680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.151+0000 7f92d6ffd700 1 --2- 172.21.15.7:0/3373069497 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92d8096a50 0x7f92d8132680 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f92c80099b0 tx=0x7f92c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d54e69dd48da5dd1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.151+0000 7f92d5ffb700 1 -- 172.21.15.7:0/3373069497 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c8009ea0 con 0x7f92d8096a50 2023-09-30T01:54:48.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.151+0000 7f92d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.151+0000 7f92d5ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:48.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.151+0000 7f92d5ffb700 10 monclient: dump: 2023-09-30T01:54:48.162 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.162 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.163 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.163 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.163 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.163 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.163 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.163 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.163 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.163 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.151+0000 7f92d5ffb700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.151+0000 7f92d5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.151+0000 7f92dd9a2700 5 monclient: authenticate success, global_id 4227 2023-09-30T01:54:48.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.151+0000 7f92dd9a2700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:48.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.151+0000 7f92d5ffb700 1 -- 172.21.15.7:0/3373069497 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c800ba30 con 0x7f92d8096a50 2023-09-30T01:54:48.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.151+0000 7f92d5ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7f92dd9a2700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:48.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7f92d5ffb700 1 -- 172.21.15.7:0/3373069497 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c800bdf0 con 0x7f92d8096a50 2023-09-30T01:54:48.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7f92d5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7f92dd9a2700 4 set_mon_vals no callback set 2023-09-30T01:54:48.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7f92d5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:48.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7f92d5ffb700 10 monclient: dump: 2023-09-30T01:54:48.164 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.164 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.164 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.165 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.165 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.165 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.165 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.165 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.165 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.165 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7fb4b37fe700 1 -- 172.21.15.7:0/4010692338 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb4b4135490 msgr2=0x7fb4b4137870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7fb4b37fe700 1 --2- 172.21.15.7:0/4010692338 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb4b4135490 0x7fb4b4137870 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009f20 tx=0x7fb4ac009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8a5da44bea5318a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7f92dd9a2700 10 monclient: shutdown 2023-09-30T01:54:48.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7f92dd9a2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7f92dd9a2700 1 -- 172.21.15.7:0/3373069497 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92d8096a50 msgr2=0x7f92d8132680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7f92dd9a2700 1 --2- 172.21.15.7:0/3373069497 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92d8096a50 0x7f92d8132680 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f92c80099b0 tx=0x7f92c8009250 comp rx=0 tx=0).stop 2023-09-30T01:54:48.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7fb4b1ffb700 1 -- 172.21.15.7:0/4010692338 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac00e070 con 0x7fb4b4135490 2023-09-30T01:54:48.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7fb4b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7fb4b1ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:54:48.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7fb4b1ffb700 10 monclient: dump: 2023-09-30T01:54:48.167 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.167 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.168 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.168 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.168 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.168 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.168 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.168 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.168 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.168 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7fb4b1ffb700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7fb4b1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7fb4b1ffb700 1 -- 172.21.15.7:0/4010692338 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ac00bc50 con 0x7fb4b4135490 2023-09-30T01:54:48.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7fb4b936f700 5 monclient: authenticate success, global_id 4226 2023-09-30T01:54:48.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7fb4b936f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:48.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7fb4b1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7fb4b936f700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:48.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7fb4b1ffb700 1 -- 172.21.15.7:0/4010692338 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac012430 con 0x7fb4b4135490 2023-09-30T01:54:48.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7fb4b936f700 4 set_mon_vals no callback set 2023-09-30T01:54:48.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7fb4b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7fb4b1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:48.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.152+0000 7fb4b1ffb700 10 monclient: dump: 2023-09-30T01:54:48.169 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.169 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.169 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.169 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.169 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.170 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.170 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.170 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.170 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.170 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.153+0000 7fb4b936f700 10 monclient: shutdown 2023-09-30T01:54:48.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.153+0000 7fb4b936f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.153+0000 7fb4b936f700 1 -- 172.21.15.7:0/4010692338 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb4b4135490 msgr2=0x7fb4b4137870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.153+0000 7fb4b936f700 1 --2- 172.21.15.7:0/4010692338 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb4b4135490 0x7fb4b4137870 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009f20 tx=0x7fb4ac009840 comp rx=0 tx=0).stop 2023-09-30T01:54:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.155+0000 7f92dd9a2700 1 -- 172.21.15.7:0/3373069497 shutdown_connections 2023-09-30T01:54:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.155+0000 7f92dd9a2700 1 --2- 172.21.15.7:0/3373069497 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92d8137710 0x7f92d8139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.155+0000 7f92dd9a2700 1 --2- 172.21.15.7:0/3373069497 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92d8096a50 0x7f92d8132680 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.155+0000 7f92dd9a2700 1 -- 172.21.15.7:0/3373069497 >> 172.21.15.7:0/3373069497 conn(0x7f92d8070cf0 msgr2=0x7f92d80710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.155+0000 7f92dd9a2700 1 -- 172.21.15.7:0/3373069497 shutdown_connections 2023-09-30T01:54:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.155+0000 7f92dd9a2700 1 -- 172.21.15.7:0/3373069497 wait complete. 2023-09-30T01:54:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.155+0000 7f92dd9a2700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.155+0000 7f92dd9a2700 10 monclient: monmap: 2023-09-30T01:54:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:48.174 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.156+0000 7f92dd9a2700 1 Processor -- start 2023-09-30T01:54:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.156+0000 7f92dd9a2700 1 -- start start 2023-09-30T01:54:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.156+0000 7f92dd9a2700 10 monclient: init 2023-09-30T01:54:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.156+0000 7f92dd9a2700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.156+0000 7f92dd9a2700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:54:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.156+0000 7f92dd9a2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92d8096a50 0x7f92d80a0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.156+0000 7f92dd9a2700 10 monclient(hunting): picked mon.noname-b con 0x7f92d8096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.156+0000 7f92dd9a2700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92d8096e20 0x7f92d809cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.156+0000 7f92dd9a2700 10 monclient(hunting): picked mon.noname-c con 0x7f92d8096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.156+0000 7f92dd9a2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92d8137710 0x7f92d809d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.156+0000 7f92dd9a2700 10 monclient(hunting): picked mon.noname-a con 0x7f92d8137710 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.156+0000 7f92dd9a2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.156+0000 7f92dd9a2700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d81334e0 con 0x7f92d8137710 2023-09-30T01:54:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.156+0000 7f92dd9a2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.156+0000 7f92dd9a2700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d8131e20 con 0x7f92d8096a50 2023-09-30T01:54:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.156+0000 7f92dd9a2700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.156+0000 7f92dd9a2700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92d80a04b0 con 0x7f92d8096e20 2023-09-30T01:54:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.156+0000 7f92dd9a2700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.156+0000 7f92dd9a2700 10 monclient(hunting): authenticate will time out at 1177.221792s 2023-09-30T01:54:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.156+0000 7f92d67fc700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92d8096e20 0x7f92d809cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.156+0000 7f92d77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92d8137710 0x7f92d809d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d77fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92d8137710 0x7f92d809d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54680/0 (socket says 172.21.15.7:54680) 2023-09-30T01:54:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d77fe700 1 -- 172.21.15.7:0/3865701853 learned_addr learned my addr 172.21.15.7:0/3865701853 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d77fe700 10 monclient(hunting): get_auth_request con 0x7f92d8137710 auth_method 0 2023-09-30T01:54:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d67fc700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92d8096e20 0x7f92d809cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59332/0 (socket says 172.21.15.7:59332) 2023-09-30T01:54:48.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d67fc700 10 monclient(hunting): get_auth_request con 0x7f92d8096e20 auth_method 0 2023-09-30T01:54:48.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d67fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d67fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d6ffd700 1 --2- 172.21.15.7:0/3865701853 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92d8096a50 0x7f92d80a0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d6ffd700 10 monclient(hunting): get_auth_request con 0x7f92d8096a50 auth_method 0 2023-09-30T01:54:48.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d67fc700 10 monclient(hunting): handle_auth_done global_id 4229 payload 824 2023-09-30T01:54:48.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d67fc700 1 -- 172.21.15.7:0/3865701853 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92d8096a50 msgr2=0x7f92d80a0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d67fc700 1 --2- 172.21.15.7:0/3865701853 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92d8096a50 0x7f92d80a0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d67fc700 1 -- 172.21.15.7:0/3865701853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92d8137710 msgr2=0x7f92d809d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d67fc700 1 --2- 172.21.15.7:0/3865701853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92d8137710 0x7f92d809d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d67fc700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d67fc700 1 monclient: found mon.noname-c 2023-09-30T01:54:48.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d67fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d67fc700 1 -- 172.21.15.7:0/3865701853 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c80094f0 con 0x7f92d8096e20 2023-09-30T01:54:48.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.157+0000 7f92d77fe700 1 --2- 172.21.15.7:0/3865701853 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92d8137710 0x7f92d809d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:48.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.158+0000 7fb4b936f700 1 -- 172.21.15.7:0/4010692338 shutdown_connections 2023-09-30T01:54:48.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.158+0000 7fb4b936f700 1 --2- 172.21.15.7:0/4010692338 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb4b4135490 0x7fb4b4137870 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.158+0000 7fb4b936f700 1 --2- 172.21.15.7:0/4010692338 >> v2:172.21.15.50:3300/0 conn(0x7fb4b4132b70 0x7fb4b4134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.158+0000 7fb4b936f700 1 -- 172.21.15.7:0/4010692338 >> 172.21.15.7:0/4010692338 conn(0x7fb4b4071910 msgr2=0x7fb4b409fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.158+0000 7fb4b936f700 1 -- 172.21.15.7:0/4010692338 shutdown_connections 2023-09-30T01:54:48.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.158+0000 7fb4b936f700 1 -- 172.21.15.7:0/4010692338 wait complete. 2023-09-30T01:54:48.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.158+0000 7fb4b936f700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.158+0000 7fb4b936f700 10 monclient: monmap: 2023-09-30T01:54:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:48.181 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7fb4b936f700 1 Processor -- start 2023-09-30T01:54:48.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7fb4b936f700 1 -- start start 2023-09-30T01:54:48.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7fb4b936f700 10 monclient: init 2023-09-30T01:54:48.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7fb4b936f700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7fb4b936f700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:54:48.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7fb4b936f700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb4b409d250 0x7fb4b413fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7fb4b936f700 10 monclient(hunting): picked mon.noname-c con 0x7fb4b409d250 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:48.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7fb4b936f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4b4132b70 0x7fb4b4140090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7fb4b936f700 10 monclient(hunting): picked mon.noname-a con 0x7fb4b4132b70 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:48.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7fb4b936f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb4b4135490 0x7fb4b411a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7fb4b936f700 10 monclient(hunting): picked mon.noname-b con 0x7fb4b4135490 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:48.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7fb4b936f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7fb4b936f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b411dda0 con 0x7fb4b4132b70 2023-09-30T01:54:48.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7fb4b936f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7fb4b936f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b409f170 con 0x7fb4b4135490 2023-09-30T01:54:48.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7fb4b936f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7fb4b936f700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b4134c20 con 0x7fb4b409d250 2023-09-30T01:54:48.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7fb4b936f700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7fb4b936f700 10 monclient(hunting): authenticate will time out at 1177.224799s 2023-09-30T01:54:48.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7f92d67fc700 1 -- 172.21.15.7:0/3865701853 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92d8096e20 msgr2=0x7f92d809cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7f92d67fc700 1 --2- 172.21.15.7:0/3865701853 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92d8096e20 0x7f92d809cb50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f92cc00b5b0 tx=0x7f92cc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.159+0000 7fb4b2ffd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb4b409d250 0x7fb4b413fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b2ffd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb4b409d250 0x7fb4b413fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59336/0 (socket says 172.21.15.7:59336) 2023-09-30T01:54:48.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b2ffd700 1 -- 172.21.15.7:0/2340587158 learned_addr learned my addr 172.21.15.7:0/2340587158 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:48.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b2ffd700 10 monclient(hunting): get_auth_request con 0x7fb4b409d250 auth_method 0 2023-09-30T01:54:48.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7f92beffd700 1 -- 172.21.15.7:0/3865701853 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc00f720 con 0x7f92d8096e20 2023-09-30T01:54:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7f92beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7f92beffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7f92beffd700 10 monclient: dump: 2023-09-30T01:54:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.185 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.185 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7f92beffd700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7f92beffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7f92beffd700 1 -- 172.21.15.7:0/3865701853 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92cc00f8c0 con 0x7f92d8096e20 2023-09-30T01:54:48.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7f92beffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7f92beffd700 1 -- 172.21.15.7:0/3865701853 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc00fc80 con 0x7f92d8096e20 2023-09-30T01:54:48.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b37fe700 1 --2- 172.21.15.7:0/2340587158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb4b4135490 0x7fb4b411a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b37fe700 10 monclient(hunting): get_auth_request con 0x7fb4b4135490 auth_method 0 2023-09-30T01:54:48.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b27fc700 1 --2- 172.21.15.7:0/2340587158 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4b4132b70 0x7fb4b4140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b27fc700 10 monclient(hunting): get_auth_request con 0x7fb4b4132b70 auth_method 0 2023-09-30T01:54:48.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b27fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b27fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7f92beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7f92beffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:48.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7f92beffd700 10 monclient: dump: 2023-09-30T01:54:48.188 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.188 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.188 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.188 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.188 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.188 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.188 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.189 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.189 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.189 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7f92d57fa700 4 set_mon_vals no callback set 2023-09-30T01:54:48.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7f92dd9a2700 5 monclient: authenticate success, global_id 4229 2023-09-30T01:54:48.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7f92dd9a2700 10 monclient: _renew_subs 2023-09-30T01:54:48.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7f92dd9a2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7f92dd9a2700 1 -- 172.21.15.7:0/3865701853 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92d809fee0 con 0x7f92d8096e20 2023-09-30T01:54:48.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7f92dd9a2700 10 monclient: _renew_subs 2023-09-30T01:54:48.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7f92dd9a2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7f92dd9a2700 1 -- 172.21.15.7:0/3865701853 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d80a13d0 con 0x7f92d8096e20 2023-09-30T01:54:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b2ffd700 10 monclient(hunting): handle_auth_done global_id 4232 payload 824 2023-09-30T01:54:48.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b2ffd700 1 -- 172.21.15.7:0/2340587158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb4b4135490 msgr2=0x7fb4b411a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b2ffd700 1 --2- 172.21.15.7:0/2340587158 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb4b4135490 0x7fb4b411a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b2ffd700 1 -- 172.21.15.7:0/2340587158 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4b4132b70 msgr2=0x7fb4b4140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b2ffd700 1 --2- 172.21.15.7:0/2340587158 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb4b4132b70 0x7fb4b4140090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b2ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b2ffd700 1 monclient: found mon.noname-c 2023-09-30T01:54:48.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b2ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.160+0000 7fb4b2ffd700 1 -- 172.21.15.7:0/2340587158 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ac009ae0 con 0x7fb4b409d250 2023-09-30T01:54:48.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.161+0000 7f92dd9a2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.161+0000 7f92dd9a2700 1 -- 172.21.15.7:0/3865701853 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f92d8064c10 con 0x7f92d8096e20 2023-09-30T01:54:48.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.162+0000 7f92beffd700 1 -- 172.21.15.7:0/3865701853 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f92cc011610 con 0x7f92d8096e20 2023-09-30T01:54:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.162+0000 7f92beffd700 1 --2- 172.21.15.7:0/3865701853 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f92c4035860 0x7f92c4037d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.162+0000 7f92beffd700 1 -- 172.21.15.7:0/3865701853 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f92cc048130 con 0x7f92d8096e20 2023-09-30T01:54:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.162+0000 7f92beffd700 1 --2- 172.21.15.7:0/3865701853 >> [v2:172.21.15.7:6816/86600,v1:172.21.15.7:6817/86600] conn(0x7f92c403b760 0x7f92c403db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.162+0000 7f92beffd700 1 -- 172.21.15.7:0/3865701853 --> [v2:172.21.15.7:6816/86600,v1:172.21.15.7:6817/86600] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92c403e220 con 0x7f92c403b760 2023-09-30T01:54:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.162+0000 7f92beffd700 1 -- 172.21.15.7:0/3865701853 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f92cc0483d0 con 0x7f92d8096e20 2023-09-30T01:54:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.162+0000 7f92beffd700 10 monclient: handle_get_version_reply finishing 1 version 15 2023-09-30T01:54:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.162+0000 7f92d6ffd700 1 --2- 172.21.15.7:0/3865701853 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f92c4035860 0x7f92c4037d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.162+0000 7f92d77fe700 1 --2- 172.21.15.7:0/3865701853 >> [v2:172.21.15.7:6816/86600,v1:172.21.15.7:6817/86600] conn(0x7f92c403b760 0x7f92c403db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.162+0000 7f92d6ffd700 10 monclient: get_auth_request con 0x7f92c4035860 auth_method 0 2023-09-30T01:54:48.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.162+0000 7f92d77fe700 10 monclient: get_auth_request con 0x7f92c403b760 auth_method 0 2023-09-30T01:54:48.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb4b2ffd700 1 -- 172.21.15.7:0/2340587158 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb4b409d250 msgr2=0x7fb4b413fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb4b2ffd700 1 --2- 172.21.15.7:0/2340587158 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb4b409d250 0x7fb4b413fb50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb4a400b600 tx=0x7fb4a40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb49affd700 1 -- 172.21.15.7:0/2340587158 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a400f720 con 0x7fb4b409d250 2023-09-30T01:54:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb49affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb49affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb49affd700 10 monclient: dump: 2023-09-30T01:54:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.196 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.197 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb49affd700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb49affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb49affd700 1 -- 172.21.15.7:0/2340587158 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a400f8a0 con 0x7fb4b409d250 2023-09-30T01:54:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb4b936f700 5 monclient: authenticate success, global_id 4232 2023-09-30T01:54:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb4b936f700 10 monclient: _renew_subs 2023-09-30T01:54:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb4b936f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb4b936f700 1 -- 172.21.15.7:0/2340587158 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b411dac0 con 0x7fb4b409d250 2023-09-30T01:54:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb49affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb49affd700 1 -- 172.21.15.7:0/2340587158 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a400fc60 con 0x7fb4b409d250 2023-09-30T01:54:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb49affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb49affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:48.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb49affd700 10 monclient: dump: 2023-09-30T01:54:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.198 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb4b936f700 10 monclient: _renew_subs 2023-09-30T01:54:48.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb4b936f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb4b936f700 1 -- 172.21.15.7:0/2340587158 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b411ec70 con 0x7fb4b409d250 2023-09-30T01:54:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.163+0000 7fb4b17fa700 4 set_mon_vals no callback set 2023-09-30T01:54:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.164+0000 7fb498ff9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.164+0000 7fb498ff9700 1 -- 172.21.15.7:0/2340587158 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb4b4140790 con 0x7fb4b409d250 2023-09-30T01:54:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.164+0000 7fb49affd700 1 -- 172.21.15.7:0/2340587158 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fb4a4011610 con 0x7fb4b409d250 2023-09-30T01:54:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.165+0000 7fb49affd700 1 --2- 172.21.15.7:0/2340587158 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb4a00357e0 0x7fb4a0037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.165+0000 7fb49affd700 1 -- 172.21.15.7:0/2340587158 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fb4a4047630 con 0x7fb4b409d250 2023-09-30T01:54:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.165+0000 7fb49affd700 1 --2- 172.21.15.7:0/2340587158 >> [v2:172.21.15.50:6804/86633,v1:172.21.15.50:6807/86633] conn(0x7fb4a003b660 0x7fb4a003da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.165+0000 7fb49affd700 1 -- 172.21.15.7:0/2340587158 --> [v2:172.21.15.50:6804/86633,v1:172.21.15.50:6807/86633] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb4a003e120 con 0x7fb4a003b660 2023-09-30T01:54:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.165+0000 7fb49affd700 1 -- 172.21.15.7:0/2340587158 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4a4049bc0 con 0x7fb4b409d250 2023-09-30T01:54:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.165+0000 7fb49affd700 10 monclient: handle_get_version_reply finishing 1 version 15 2023-09-30T01:54:48.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.165+0000 7fb4b27fc700 1 --2- 172.21.15.7:0/2340587158 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb4a00357e0 0x7fb4a0037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.165+0000 7fb4b27fc700 10 monclient: get_auth_request con 0x7fb4a00357e0 auth_method 0 2023-09-30T01:54:48.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.165+0000 7fb4b37fe700 1 --2- 172.21.15.7:0/2340587158 >> [v2:172.21.15.50:6804/86633,v1:172.21.15.50:6807/86633] conn(0x7fb4a003b660 0x7fb4a003da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.165+0000 7fb4b37fe700 10 monclient: get_auth_request con 0x7fb4a003b660 auth_method 0 2023-09-30T01:54:48.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.167+0000 7f92d77fe700 1 --2- 172.21.15.7:0/3865701853 >> [v2:172.21.15.7:6816/86600,v1:172.21.15.7:6817/86600] conn(0x7f92c403b760 0x7f92c403db70 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.167+0000 7fb4b27fc700 1 --2- 172.21.15.7:0/2340587158 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb4a00357e0 0x7fb4a0037c90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8009bf0 tx=0x7fb4a80091a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.168+0000 7f92beffd700 1 -- 172.21.15.7:0/3865701853 <== osd.1 v2:172.21.15.7:6816/86600 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f92c403e220 con 0x7f92c403b760 2023-09-30T01:54:48.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.170+0000 7f92d6ffd700 1 --2- 172.21.15.7:0/3865701853 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f92c4035860 0x7f92c4037d10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f92c80099b0 tx=0x7f92c8016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.170+0000 7fb4b37fe700 1 --2- 172.21.15.7:0/2340587158 >> [v2:172.21.15.50:6804/86633,v1:172.21.15.50:6807/86633] conn(0x7fb4a003b660 0x7fb4a003da70 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.170+0000 7fb49affd700 1 -- 172.21.15.7:0/2340587158 <== osd.3 v2:172.21.15.50:6804/86633 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fb4a003e120 con 0x7fb4a003b660 2023-09-30T01:54:48.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.186+0000 7fb498ff9700 1 -- 172.21.15.7:0/2340587158 --> [v2:172.21.15.50:6804/86633,v1:172.21.15.50:6807/86633] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb4b405e280 con 0x7fb4a003b660 2023-09-30T01:54:48.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.187+0000 7fb49affd700 1 -- 172.21.15.7:0/2340587158 <== osd.3 v2:172.21.15.50:6804/86633 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb4b405e280 con 0x7fb4a003b660 2023-09-30T01:54:48.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.187+0000 7fb498ff9700 1 -- 172.21.15.7:0/2340587158 >> [v2:172.21.15.50:6804/86633,v1:172.21.15.50:6807/86633] conn(0x7fb4a003b660 msgr2=0x7fb4a003da70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.187+0000 7fb498ff9700 1 --2- 172.21.15.7:0/2340587158 >> [v2:172.21.15.50:6804/86633,v1:172.21.15.50:6807/86633] conn(0x7fb4a003b660 0x7fb4a003da70 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.188+0000 7fb498ff9700 1 -- 172.21.15.7:0/2340587158 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb4a00357e0 msgr2=0x7fb4a0037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.188+0000 7fb498ff9700 1 --2- 172.21.15.7:0/2340587158 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb4a00357e0 0x7fb4a0037c90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8009bf0 tx=0x7fb4a80091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.188+0000 7fb498ff9700 10 monclient: shutdown 2023-09-30T01:54:48.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.188+0000 7fb498ff9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.188+0000 7fb498ff9700 1 -- 172.21.15.7:0/2340587158 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb4b409d250 msgr2=0x7fb4b413fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.188+0000 7fb498ff9700 1 --2- 172.21.15.7:0/2340587158 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb4b409d250 0x7fb4b413fb50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb4a400b600 tx=0x7fb4a40091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.188+0000 7fb4b37fe700 1 -- 172.21.15.7:0/2340587158 reap_dead start 2023-09-30T01:54:48.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.191+0000 7fb498ff9700 1 -- 172.21.15.7:0/2340587158 shutdown_connections 2023-09-30T01:54:48.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.191+0000 7fb498ff9700 1 -- 172.21.15.7:0/2340587158 >> 172.21.15.7:0/2340587158 conn(0x7fb4b4071910 msgr2=0x7fb4b41337d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.191+0000 7fb498ff9700 1 -- 172.21.15.7:0/2340587158 shutdown_connections 2023-09-30T01:54:48.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.191+0000 7fb498ff9700 1 -- 172.21.15.7:0/2340587158 wait complete. 2023-09-30T01:54:48.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.196+0000 7f92dd9a2700 1 -- 172.21.15.7:0/3865701853 --> [v2:172.21.15.7:6816/86600,v1:172.21.15.7:6817/86600] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f92d80a2440 con 0x7f92c403b760 2023-09-30T01:54:48.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.196+0000 7f92beffd700 1 -- 172.21.15.7:0/3865701853 <== osd.1 v2:172.21.15.7:6816/86600 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f92d80a2440 con 0x7f92c403b760 2023-09-30T01:54:48.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.197+0000 7f92dd9a2700 1 -- 172.21.15.7:0/3865701853 >> [v2:172.21.15.7:6816/86600,v1:172.21.15.7:6817/86600] conn(0x7f92c403b760 msgr2=0x7f92c403db70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.197+0000 7f92dd9a2700 1 --2- 172.21.15.7:0/3865701853 >> [v2:172.21.15.7:6816/86600,v1:172.21.15.7:6817/86600] conn(0x7f92c403b760 0x7f92c403db70 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.197+0000 7f92dd9a2700 1 -- 172.21.15.7:0/3865701853 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f92c4035860 msgr2=0x7f92c4037d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.197+0000 7f92dd9a2700 1 --2- 172.21.15.7:0/3865701853 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f92c4035860 0x7f92c4037d10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f92c80099b0 tx=0x7f92c8016000 comp rx=0 tx=0).stop 2023-09-30T01:54:48.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.197+0000 7f92dd9a2700 10 monclient: shutdown 2023-09-30T01:54:48.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.197+0000 7f92dd9a2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.197+0000 7f92dd9a2700 1 -- 172.21.15.7:0/3865701853 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92d8096e20 msgr2=0x7f92d809cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.197+0000 7f92dd9a2700 1 --2- 172.21.15.7:0/3865701853 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92d8096e20 0x7f92d809cb50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f92cc00b5b0 tx=0x7f92cc0090f0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.197+0000 7f92d77fe700 1 -- 172.21.15.7:0/3865701853 reap_dead start 2023-09-30T01:54:48.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.199+0000 7f8a6141d700 10 monclient: get_monmap_and_config 2023-09-30T01:54:48.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.199+0000 7f8a6141d700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.199+0000 7f8a6141d700 10 monclient: monmap: 2023-09-30T01:54:48.206 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.207 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.207 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:48.200683+0000 2023-09-30T01:54:48.207 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:48.200683+0000 2023-09-30T01:54:48.207 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.207 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.207 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:48.207 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:48.207 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:48.207 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:48.207 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:48.207 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.199+0000 7f08817de700 10 monclient: get_monmap_and_config 2023-09-30T01:54:48.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.199+0000 7f08817de700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.199+0000 7f08817de700 10 monclient: monmap: 2023-09-30T01:54:48.208 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.208 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.208 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:48.200958+0000 2023-09-30T01:54:48.208 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:48.200958+0000 2023-09-30T01:54:48.208 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.209 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.209 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:48.209 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:48.209 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:48.209 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:48.209 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:48.209 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.200+0000 7f92dd9a2700 1 -- 172.21.15.7:0/3865701853 shutdown_connections 2023-09-30T01:54:48.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.200+0000 7f92dd9a2700 1 -- 172.21.15.7:0/3865701853 >> 172.21.15.7:0/3865701853 conn(0x7f92d8070cf0 msgr2=0x7f92d812f110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.200+0000 7f92dd9a2700 1 -- 172.21.15.7:0/3865701853 shutdown_connections 2023-09-30T01:54:48.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.200+0000 7f92dd9a2700 1 -- 172.21.15.7:0/3865701853 wait complete. 2023-09-30T01:54:48.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.200+0000 7f8a6141d700 1 Processor -- start 2023-09-30T01:54:48.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.200+0000 7f8a6141d700 1 -- start start 2023-09-30T01:54:48.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.200+0000 7f8a6141d700 10 monclient: init 2023-09-30T01:54:48.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a6141d700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a6141d700 10 monclient: _add_conns ranks=[4,3,1,0,2] 2023-09-30T01:54:48.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a6141d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8a5c096a50 0x7f8a5c132680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a6141d700 10 monclient(hunting): picked mon.noname-c con 0x7f8a5c096a50 addr v2:172.21.15.50:3300/0 2023-09-30T01:54:48.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a6141d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8a5c096e20 0x7f8a5c097220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a6141d700 10 monclient(hunting): picked mon.noname-a con 0x7f8a5c096e20 addr v2:172.21.15.7:3300/0 2023-09-30T01:54:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a6141d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8a5c12ef00 addr v1:172.21.15.50:6789/0 2023-09-30T01:54:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a6141d700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a5c097b40 con 0x7f8a5c12ef00 2023-09-30T01:54:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a6141d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a6141d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a5c099160 con 0x7f8a5c096e20 2023-09-30T01:54:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a6141d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a6141d700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a5c099720 con 0x7f8a5c096a50 2023-09-30T01:54:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a6141d700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a6141d700 10 monclient(hunting): authenticate will time out at 1177.266373s 2023-09-30T01:54:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a5a7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8a5c096e20 0x7f8a5c097220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a5a7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8a5c096e20 0x7f8a5c097220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54704/0 (socket says 172.21.15.7:54704) 2023-09-30T01:54:48.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a5a7fc700 1 -- 172.21.15.7:0/1234032201 learned_addr learned my addr 172.21.15.7:0/1234032201 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:48.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a5a7fc700 10 monclient(hunting): get_auth_request con 0x7f8a5c096e20 auth_method 0 2023-09-30T01:54:48.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a5a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a5a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a5a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a5affd700 1 --2- 172.21.15.7:0/1234032201 >> v2:172.21.15.50:3300/0 conn(0x7f8a5c096a50 0x7f8a5c132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a5affd700 10 monclient(hunting): get_auth_request con 0x7f8a5c096a50 auth_method 0 2023-09-30T01:54:48.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a5affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a5affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a5affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a5affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a5affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.201+0000 7f8a5affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.202+0000 7f8a5affd700 10 monclient(hunting): handle_auth_done global_id 4234 payload 274 2023-09-30T01:54:48.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.202+0000 7f8a5affd700 1 -- 172.21.15.7:0/1234032201 >> v2:172.21.15.7:3300/0 conn(0x7f8a5c096e20 msgr2=0x7f8a5c097220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.202+0000 7f8a5affd700 1 --2- 172.21.15.7:0/1234032201 >> v2:172.21.15.7:3300/0 conn(0x7f8a5c096e20 0x7f8a5c097220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.202+0000 7f8a5affd700 1 -- 172.21.15.7:0/1234032201 >> v1:172.21.15.50:6789/0 conn(0x7f8a5c12ef00 legacy=0x7f8a5c12f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.202+0000 7f8a5affd700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.202+0000 7f8a5affd700 1 monclient: found mon.noname-c 2023-09-30T01:54:48.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.202+0000 7f8a5affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.202+0000 7f8a5affd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3300/0 2023-09-30T01:54:48.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.202+0000 7f8a5affd700 1 -- 172.21.15.7:0/1234032201 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c097760 con 0x7f8a5c096a50 2023-09-30T01:54:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.202+0000 7f6ae4774700 10 monclient: get_monmap_and_config 2023-09-30T01:54:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.202+0000 7f6ae4774700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.202+0000 7f6ae4774700 10 monclient: monmap: 2023-09-30T01:54:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:48.203096+0000 2023-09-30T01:54:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:48.203096+0000 2023-09-30T01:54:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:48.217 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:48.218 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.202+0000 7f6ae4774700 1 Processor -- start 2023-09-30T01:54:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.202+0000 7f6ae4774700 1 -- start start 2023-09-30T01:54:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.202+0000 7f6ae4774700 10 monclient: init 2023-09-30T01:54:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.203+0000 7f6ae4774700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.203+0000 7f6ae4774700 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-09-30T01:54:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.203+0000 7f6ae4774700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6adc132310 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.203+0000 7f6ae4774700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6adc132df0 addr v1:172.21.15.50:6789/0 2023-09-30T01:54:48.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.203+0000 7f6ae4774700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6adc06afc0 0x7f6adc13b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.203+0000 7f6ae4774700 10 monclient(hunting): picked mon.noname-c con 0x7f6adc06afc0 addr v2:172.21.15.50:3300/0 2023-09-30T01:54:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.203+0000 7f6ae4774700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6adc06a310 con 0x7f6adc132310 2023-09-30T01:54:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.203+0000 7f6ae4774700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6adc06b3c0 con 0x7f6adc132df0 2023-09-30T01:54:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.203+0000 7f6ae4774700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.203+0000 7f6ae4774700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6adc069830 con 0x7f6adc06afc0 2023-09-30T01:54:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.203+0000 7f6ae4774700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.203+0000 7f6ae4774700 10 monclient(hunting): authenticate will time out at 1177.268485s 2023-09-30T01:54:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.203+0000 7f6ae1d0f700 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f6adc132df0 0x7f6adc1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.7:54066/0 (socket says 172.21.15.7:54066) 2023-09-30T01:54:48.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.203+0000 7f6ae1d0f700 1 -- 172.21.15.7:0/1183618987 learned_addr learned my addr 172.21.15.7:0/1183618987 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:54:48.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f8a5affd700 1 -- 172.21.15.7:0/1234032201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a5c096a50 msgr2=0x7f8a5c132680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f8a5affd700 1 --2- 172.21.15.7:0/1234032201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a5c096a50 0x7f8a5c132680 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c0a2b30 tx=0x7f8a500091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d9dab573e41add3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f6ae2d11700 1 --2- 172.21.15.7:0/1183618987 >> v2:172.21.15.50:3300/0 conn(0x7f6adc06afc0 0x7f6adc13b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f6ae2d11700 10 monclient(hunting): get_auth_request con 0x7f6adc06afc0 auth_method 0 2023-09-30T01:54:48.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f6ae2d11700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f6ae2d11700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f6ae2d11700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f6ae2d11700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f6ae2d11700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f6ae2d11700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f8a59ffb700 1 -- 172.21.15.7:0/1234032201 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5000b880 con 0x7f8a5c096a50 2023-09-30T01:54:48.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f8a59ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f8a59ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:48.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f8a59ffb700 10 monclient: dump: 2023-09-30T01:54:48.222 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.222 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.223 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.223 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.223 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.223 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.223 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.223 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.223 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.223 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f8a59ffb700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f8a59ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f8a59ffb700 1 -- 172.21.15.7:0/1234032201 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a5000b9e0 con 0x7f8a5c096a50 2023-09-30T01:54:48.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f8a6141d700 5 monclient: authenticate success, global_id 4234 2023-09-30T01:54:48.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f8a6141d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:48.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f8a59ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f8a6141d700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:48.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f8a59ffb700 1 -- 172.21.15.7:0/1234032201 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5000bd80 con 0x7f8a5c096a50 2023-09-30T01:54:48.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f8a59ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f8a6141d700 4 set_mon_vals no callback set 2023-09-30T01:54:48.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f8a59ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:54:48.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f8a59ffb700 10 monclient: dump: 2023-09-30T01:54:48.224 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.224 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.224 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.225 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.225 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.225 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.225 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.225 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.225 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.225 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f6ae2d11700 10 monclient(hunting): handle_auth_done global_id 4237 payload 274 2023-09-30T01:54:48.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f6ae2d11700 1 -- 172.21.15.7:0/1183618987 >> v1:172.21.15.50:6789/0 conn(0x7f6adc132df0 legacy=0x7f6adc1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f6ae2d11700 1 -- 172.21.15.7:0/1183618987 >> v1:172.21.15.7:6789/0 conn(0x7f6adc132310 legacy=0x7f6adc1326e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f6ae2d11700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f6ae2d11700 1 monclient: found mon.noname-c 2023-09-30T01:54:48.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f6ae2d11700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f6ae2d11700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3300/0 2023-09-30T01:54:48.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.204+0000 7f6ae2d11700 1 -- 172.21.15.7:0/1183618987 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6adc06a030 con 0x7f6adc06afc0 2023-09-30T01:54:48.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.205+0000 7f8a6141d700 10 monclient: shutdown 2023-09-30T01:54:48.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.205+0000 7f8a6141d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.205+0000 7f8a6141d700 1 -- 172.21.15.7:0/1234032201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a5c096a50 msgr2=0x7f8a5c132680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.205+0000 7f8a6141d700 1 --2- 172.21.15.7:0/1234032201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a5c096a50 0x7f8a5c132680 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c0a2b30 tx=0x7f8a500091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.205+0000 7f8a6141d700 1 -- 172.21.15.7:0/1234032201 shutdown_connections 2023-09-30T01:54:48.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.205+0000 7f8a6141d700 1 --2- 172.21.15.7:0/1234032201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a5c096a50 0x7f8a5c132680 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.205+0000 7f8a6141d700 1 --2- 172.21.15.7:0/1234032201 >> v2:172.21.15.7:3300/0 conn(0x7f8a5c096e20 0x7f8a5c097220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.205+0000 7f8a6141d700 1 -- 172.21.15.7:0/1234032201 >> 172.21.15.7:0/1234032201 conn(0x7f8a5c070cf0 msgr2=0x7f8a5c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.205+0000 7f8a6141d700 1 -- 172.21.15.7:0/1234032201 shutdown_connections 2023-09-30T01:54:48.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.205+0000 7f8a6141d700 1 -- 172.21.15.7:0/1234032201 wait complete. 2023-09-30T01:54:48.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.205+0000 7f08817de700 1 Processor -- start 2023-09-30T01:54:48.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f08817de700 1 -- start start 2023-09-30T01:54:48.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f08817de700 10 monclient: init 2023-09-30T01:54:48.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f4457da8700 10 monclient: get_monmap_and_config 2023-09-30T01:54:48.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f4457da8700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f4457da8700 10 monclient: monmap: 2023-09-30T01:54:48.229 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.229 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.229 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:48.207104+0000 2023-09-30T01:54:48.229 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:48.207104+0000 2023-09-30T01:54:48.229 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.229 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.229 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:48.229 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:48.230 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:48.230 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:48.230 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:48.230 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f8a6141d700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f8a6141d700 10 monclient: monmap: 2023-09-30T01:54:48.230 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.230 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:48.231 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:48.231 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:48.231 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.231 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.231 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:48.231 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:48.231 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:48.231 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f4457da8700 1 Processor -- start 2023-09-30T01:54:48.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f08817de700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f08817de700 10 monclient: _add_conns ranks=[2,3,0,4,1] 2023-09-30T01:54:48.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f08817de700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f087c06f510 0x7f087c09b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f08817de700 10 monclient(hunting): picked mon.noname-b con 0x7f087c06f510 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:48.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f08817de700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f087c069b70 0x7f087c09dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f08817de700 10 monclient(hunting): picked mon.noname-a con 0x7f087c069b70 addr v2:172.21.15.7:3300/0 2023-09-30T01:54:48.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f08817de700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f087c06f8e0 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:48.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f08817de700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087c069950 con 0x7f087c06f8e0 2023-09-30T01:54:48.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f08817de700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f08817de700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f087c06b1c0 con 0x7f087c069b70 2023-09-30T01:54:48.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f08817de700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f08817de700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087c09b630 con 0x7f087c06f510 2023-09-30T01:54:48.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f08817de700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f08817de700 10 monclient(hunting): authenticate will time out at 1177.271887s 2023-09-30T01:54:48.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f8a6141d700 1 Processor -- start 2023-09-30T01:54:48.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f6ae2d11700 1 -- 172.21.15.7:0/1183618987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6adc06afc0 msgr2=0x7f6adc13b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f6ae2d11700 1 --2- 172.21.15.7:0/1183618987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6adc06afc0 0x7f6adc13b000 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4009e60 tx=0x7f6ad4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34bfb195d749d0c4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087bfff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f087c06f510 0x7f087c09b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087bfff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f087c06f510 0x7f087c09b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59342/0 (socket says 172.21.15.7:59342) 2023-09-30T01:54:48.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087bfff700 1 -- 172.21.15.7:0/4220366267 learned_addr learned my addr 172.21.15.7:0/4220366267 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:48.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087bfff700 10 monclient(hunting): get_auth_request con 0x7f087c06f510 auth_method 0 2023-09-30T01:54:48.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087b7fe700 1 --2- 172.21.15.7:0/4220366267 >> v2:172.21.15.7:3300/0 conn(0x7f087c069b70 0x7f087c09dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087b7fe700 10 monclient(hunting): get_auth_request con 0x7f087c069b70 auth_method 0 2023-09-30T01:54:48.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087bfff700 10 monclient(hunting): handle_auth_done global_id 4235 payload 274 2023-09-30T01:54:48.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087bfff700 1 -- 172.21.15.7:0/4220366267 >> v2:172.21.15.7:3300/0 conn(0x7f087c069b70 msgr2=0x7f087c09dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087bfff700 1 --2- 172.21.15.7:0/4220366267 >> v2:172.21.15.7:3300/0 conn(0x7f087c069b70 0x7f087c09dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087bfff700 1 -- 172.21.15.7:0/4220366267 >> v1:172.21.15.7:6789/0 conn(0x7f087c06f8e0 legacy=0x7f087c09e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087bfff700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087bfff700 1 monclient: found mon.noname-b 2023-09-30T01:54:48.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f087bfff700 1 -- 172.21.15.7:0/4220366267 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087c06aee0 con 0x7f087c06f510 2023-09-30T01:54:48.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f4457da8700 1 -- start start 2023-09-30T01:54:48.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.206+0000 7f4457da8700 10 monclient: init 2023-09-30T01:54:48.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4457da8700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4457da8700 10 monclient: _add_conns ranks=[3,2,0,4,1] 2023-09-30T01:54:48.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4457da8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f445009c520 0x7f445009c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4457da8700 10 monclient(hunting): picked mon.noname-a con 0x7f445009c520 addr v2:172.21.15.7:3300/0 2023-09-30T01:54:48.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4457da8700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f445009bcb0 0x7f445009ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4457da8700 10 monclient(hunting): picked mon.noname-b con 0x7f445009bcb0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:48.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4457da8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f445013ab20 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:48.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4457da8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f445009d370 con 0x7f445013ab20 2023-09-30T01:54:48.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4457da8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4457da8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f445006b870 con 0x7f445009c520 2023-09-30T01:54:48.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4457da8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4457da8700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f445006b260 con 0x7f445009bcb0 2023-09-30T01:54:48.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4457da8700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4457da8700 10 monclient(hunting): authenticate will time out at 1177.272714s 2023-09-30T01:54:48.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4455b44700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f445009c520 0x7f445009c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4455343700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f445009bcb0 0x7f445009ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4455b44700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f445009c520 0x7f445009c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54720/0 (socket says 172.21.15.7:54720) 2023-09-30T01:54:48.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4455343700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f445009bcb0 0x7f445009ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59356/0 (socket says 172.21.15.7:59356) 2023-09-30T01:54:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4455b44700 1 -- 172.21.15.7:0/2117379672 learned_addr learned my addr 172.21.15.7:0/2117379672 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4455b44700 10 monclient(hunting): get_auth_request con 0x7f445009c520 auth_method 0 2023-09-30T01:54:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4455b44700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4455b44700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4455b44700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4455343700 10 monclient(hunting): get_auth_request con 0x7f445009bcb0 auth_method 0 2023-09-30T01:54:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4455343700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4455343700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4455343700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4455343700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f4455343700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f4455343700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f4455b44700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f4455b44700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f4455b44700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f4455343700 10 monclient(hunting): handle_auth_done global_id 4238 payload 274 2023-09-30T01:54:48.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f4455343700 1 -- 172.21.15.7:0/2117379672 >> v2:172.21.15.7:3300/0 conn(0x7f445009c520 msgr2=0x7f445009c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f4455343700 1 --2- 172.21.15.7:0/2117379672 >> v2:172.21.15.7:3300/0 conn(0x7f445009c520 0x7f445009c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.242 INFO:teuthology.orchestra.run.smithi007.stdout:47244640258 2023-09-30T01:54:48.242 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T01:54:48.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f4455343700 1 -- 172.21.15.7:0/2117379672 >> v1:172.21.15.7:6789/0 conn(0x7f445013ab20 legacy=0x7f445013cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f4455343700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f4455343700 1 monclient: found mon.noname-b 2023-09-30T01:54:48.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f4455343700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f4455343700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f4455343700 1 -- 172.21.15.7:0/2117379672 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f445006b590 con 0x7f445009bcb0 2023-09-30T01:54:48.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f8a6141d700 1 -- start start 2023-09-30T01:54:48.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.207+0000 7f8a6141d700 10 monclient: init 2023-09-30T01:54:48.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a6141d700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a6141d700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:54:48.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a6141d700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8a5c096a50 0x7f8a5c06e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a6141d700 10 monclient(hunting): picked mon.noname-c con 0x7f8a5c096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:48.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a6141d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a5c096e20 0x7f8a5c0a0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a6141d700 10 monclient(hunting): picked mon.noname-a con 0x7f8a5c096e20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:48.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a6141d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a5c12ef00 0x7f8a5c06b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a6141d700 10 monclient(hunting): picked mon.noname-b con 0x7f8a5c12ef00 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:48.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a6141d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a6141d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c06f4a0 con 0x7f8a5c096e20 2023-09-30T01:54:48.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a6141d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a6141d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c0a13d0 con 0x7f8a5c12ef00 2023-09-30T01:54:48.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a6141d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a6141d700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c1334e0 con 0x7f8a5c096a50 2023-09-30T01:54:48.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a6141d700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a6141d700 10 monclient(hunting): authenticate will time out at 1177.273199s 2023-09-30T01:54:48.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5affd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8a5c096a50 0x7f8a5c06e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5affd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8a5c096a50 0x7f8a5c06e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59370/0 (socket says 172.21.15.7:59370) 2023-09-30T01:54:48.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5affd700 1 -- 172.21.15.7:0/3745188111 learned_addr learned my addr 172.21.15.7:0/3745188111 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:48.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5affd700 10 monclient(hunting): get_auth_request con 0x7f8a5c096a50 auth_method 0 2023-09-30T01:54:48.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f6ae150e700 1 -- 172.21.15.7:0/1183618987 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad400e070 con 0x7f6adc06afc0 2023-09-30T01:54:48.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f6ae150e700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f6ae150e700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f6ae150e700 10 monclient: dump: 2023-09-30T01:54:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.247 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f6ae150e700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5a7fc700 1 --2- 172.21.15.7:0/3745188111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a5c096e20 0x7f8a5c0a0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f6ae150e700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f6ae4774700 5 monclient: authenticate success, global_id 4237 2023-09-30T01:54:48.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f6ae150e700 1 -- 172.21.15.7:0/1183618987 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad4012430 con 0x7f6adc06afc0 2023-09-30T01:54:48.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f6ae4774700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:48.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f6ae150e700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f6ae150e700 1 -- 172.21.15.7:0/1183618987 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad400bea0 con 0x7f6adc06afc0 2023-09-30T01:54:48.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5a7fc700 10 monclient(hunting): get_auth_request con 0x7f8a5c096e20 auth_method 0 2023-09-30T01:54:48.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f6ae150e700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f6ae150e700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:54:48.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f6ae150e700 10 monclient: dump: 2023-09-30T01:54:48.250 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.250 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.250 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.250 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.250 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.250 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.250 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.250 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.250 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.251 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f6ae4774700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:48.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f6ae4774700 4 set_mon_vals no callback set 2023-09-30T01:54:48.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5b7fe700 1 --2- 172.21.15.7:0/3745188111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a5c12ef00 0x7f8a5c06b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5b7fe700 10 monclient(hunting): get_auth_request con 0x7f8a5c12ef00 auth_method 0 2023-09-30T01:54:48.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5affd700 10 monclient(hunting): handle_auth_done global_id 4241 payload 824 2023-09-30T01:54:48.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5affd700 1 -- 172.21.15.7:0/3745188111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a5c12ef00 msgr2=0x7f8a5c06b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5affd700 1 --2- 172.21.15.7:0/3745188111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8a5c12ef00 0x7f8a5c06b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5affd700 1 -- 172.21.15.7:0/3745188111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a5c096e20 msgr2=0x7f8a5c0a0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5affd700 1 --2- 172.21.15.7:0/3745188111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a5c096e20 0x7f8a5c0a0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5affd700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5affd700 1 monclient: found mon.noname-c 2023-09-30T01:54:48.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5affd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.208+0000 7f8a5affd700 1 -- 172.21.15.7:0/3745188111 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a50009420 con 0x7f8a5c096a50 2023-09-30T01:54:48.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.209+0000 7f6ae4774700 10 monclient: shutdown 2023-09-30T01:54:48.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.209+0000 7f6ae4774700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.209+0000 7f6ae4774700 1 -- 172.21.15.7:0/1183618987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6adc06afc0 msgr2=0x7f6adc13b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.209+0000 7f6ae4774700 1 --2- 172.21.15.7:0/1183618987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6adc06afc0 0x7f6adc13b000 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4009e60 tx=0x7f6ad4009780 comp rx=0 tx=0).stop 2023-09-30T01:54:48.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.209+0000 7f6ae4774700 1 -- 172.21.15.7:0/1183618987 shutdown_connections 2023-09-30T01:54:48.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.209+0000 7f6ae4774700 1 --2- 172.21.15.7:0/1183618987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6adc06afc0 0x7f6adc13b000 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.209+0000 7f6ae4774700 1 -- 172.21.15.7:0/1183618987 >> 172.21.15.7:0/1183618987 conn(0x7f6adc067f30 msgr2=0x7f6adc09fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.209+0000 7f6ae4774700 1 -- 172.21.15.7:0/1183618987 shutdown_connections 2023-09-30T01:54:48.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.209+0000 7f6ae4774700 1 -- 172.21.15.7:0/1183618987 wait complete. 2023-09-30T01:54:48.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.209+0000 7f087bfff700 1 -- 172.21.15.7:0/4220366267 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f087c06f510 msgr2=0x7f087c09b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.209+0000 7f087bfff700 1 --2- 172.21.15.7:0/4220366267 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f087c06f510 0x7f087c09b010 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f086c0099b0 tx=0x7f086c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11693a6fe15287e3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f6ae4774700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f6ae4774700 10 monclient: monmap: 2023-09-30T01:54:48.255 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.255 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:48.255 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:48.255 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:48.255 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.256 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.256 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:48.256 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:48.256 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:48.256 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f087affd700 1 -- 172.21.15.7:0/4220366267 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c009ea0 con 0x7f087c06f510 2023-09-30T01:54:48.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f087affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f087affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:54:48.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f087affd700 10 monclient: dump: 2023-09-30T01:54:48.257 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.257 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.257 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.257 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.257 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.257 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.257 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.257 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.257 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.258 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f087affd700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f087affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f08817de700 5 monclient: authenticate success, global_id 4235 2023-09-30T01:54:48.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f08817de700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:48.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f087affd700 1 -- 172.21.15.7:0/4220366267 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086c00ba30 con 0x7f087c06f510 2023-09-30T01:54:48.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f087affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f08817de700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:48.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f087affd700 1 -- 172.21.15.7:0/4220366267 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c00bdf0 con 0x7f087c06f510 2023-09-30T01:54:48.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f087affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f08817de700 4 set_mon_vals no callback set 2023-09-30T01:54:48.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f087affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:48.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f087affd700 10 monclient: dump: 2023-09-30T01:54:48.259 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.259 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.259 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.259 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.259 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.259 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.259 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.259 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.259 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.259 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f6ae4774700 1 Processor -- start 2023-09-30T01:54:48.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f4455343700 1 -- 172.21.15.7:0/2117379672 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f445009bcb0 msgr2=0x7f445009ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f4455343700 1 --2- 172.21.15.7:0/2117379672 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f445009bcb0 0x7f445009ce30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f44400098e0 tx=0x7f44400091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a939e9b40d5764ce server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f6ae4774700 1 -- start start 2023-09-30T01:54:48.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f6ae4774700 10 monclient: init 2023-09-30T01:54:48.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f4454b42700 1 -- 172.21.15.7:0/2117379672 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4440011840 con 0x7f445009bcb0 2023-09-30T01:54:48.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f4454b42700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f4454b42700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:54:48.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f4454b42700 10 monclient: dump: 2023-09-30T01:54:48.262 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.262 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.262 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.262 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.262 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.262 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.262 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.262 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.263 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.263 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f4454b42700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f4454b42700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f4454b42700 1 -- 172.21.15.7:0/2117379672 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44400119a0 con 0x7f445009bcb0 2023-09-30T01:54:48.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f4457da8700 5 monclient: authenticate success, global_id 4238 2023-09-30T01:54:48.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f4457da8700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:48.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f4454b42700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f4457da8700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:48.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f4454b42700 1 -- 172.21.15.7:0/2117379672 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4440011d60 con 0x7f445009bcb0 2023-09-30T01:54:48.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f4454b42700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f4457da8700 4 set_mon_vals no callback set 2023-09-30T01:54:48.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f4454b42700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:48.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.210+0000 7f4454b42700 10 monclient: dump: 2023-09-30T01:54:48.264 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.264 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.264 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.264 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.264 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.264 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.264 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.264 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.264 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.264 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a5affd700 1 -- 172.21.15.7:0/3745188111 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8a5c096a50 msgr2=0x7f8a5c06e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a5affd700 1 --2- 172.21.15.7:0/3745188111 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8a5c096a50 0x7f8a5c06e9e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8a5000b0e0 tx=0x7f8a50016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f08817de700 10 monclient: shutdown 2023-09-30T01:54:48.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f08817de700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f08817de700 1 -- 172.21.15.7:0/4220366267 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f087c06f510 msgr2=0x7f087c09b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f08817de700 1 --2- 172.21.15.7:0/4220366267 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f087c06f510 0x7f087c09b010 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f086c0099b0 tx=0x7f086c009250 comp rx=0 tx=0).stop 2023-09-30T01:54:48.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae4774700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae4774700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:54:48.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae4774700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6adc06afc0 0x7f6adc141da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae4774700 10 monclient(hunting): picked mon.noname-b con 0x7f6adc06afc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:48.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae4774700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6adc132310 0x7f6adc12f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae4774700 10 monclient(hunting): picked mon.noname-c con 0x7f6adc132310 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:48.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae4774700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6adc132df0 0x7f6adc0986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae4774700 10 monclient(hunting): picked mon.noname-a con 0x7f6adc132df0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:48.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae4774700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae4774700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc11f9d0 con 0x7f6adc132df0 2023-09-30T01:54:48.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae4774700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae4774700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc11e380 con 0x7f6adc06afc0 2023-09-30T01:54:48.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae4774700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae4774700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc11f6f0 con 0x7f6adc132310 2023-09-30T01:54:48.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae4774700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae4774700 10 monclient(hunting): authenticate will time out at 1177.276236s 2023-09-30T01:54:48.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a42ffd700 1 -- 172.21.15.7:0/3745188111 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a50006b10 con 0x7f8a5c096a50 2023-09-30T01:54:48.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a42ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a42ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:48.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a42ffd700 10 monclient: dump: 2023-09-30T01:54:48.271 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.271 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.271 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.271 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.272 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.272 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.272 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.272 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.272 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.272 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a42ffd700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a42ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a42ffd700 1 -- 172.21.15.7:0/3745188111 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a50011db0 con 0x7f8a5c096a50 2023-09-30T01:54:48.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a42ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a42ffd700 1 -- 172.21.15.7:0/3745188111 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5001a6b0 con 0x7f8a5c096a50 2023-09-30T01:54:48.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a42ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a42ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:48.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a42ffd700 10 monclient: dump: 2023-09-30T01:54:48.273 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.273 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.273 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.273 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.273 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.273 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.273 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.273 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.274 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.274 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a6141d700 5 monclient: authenticate success, global_id 4241 2023-09-30T01:54:48.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a6141d700 10 monclient: _renew_subs 2023-09-30T01:54:48.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a6141d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a6141d700 1 -- 172.21.15.7:0/3745188111 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a5c135710 con 0x7f8a5c096a50 2023-09-30T01:54:48.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a597fa700 4 set_mon_vals no callback set 2023-09-30T01:54:48.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae2d11700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6adc132df0 0x7f6adc0986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae2d11700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6adc132df0 0x7f6adc0986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54732/0 (socket says 172.21.15.7:54732) 2023-09-30T01:54:48.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae2d11700 1 -- 172.21.15.7:0/3161612665 learned_addr learned my addr 172.21.15.7:0/3161612665 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:48.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae2d11700 10 monclient(hunting): get_auth_request con 0x7f6adc132df0 auth_method 0 2023-09-30T01:54:48.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae2d11700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae2d11700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae2d11700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae1d0f700 1 --2- 172.21.15.7:0/3161612665 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6adc132310 0x7f6adc12f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae1d0f700 10 monclient(hunting): get_auth_request con 0x7f6adc132310 auth_method 0 2023-09-30T01:54:48.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae1d0f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae1d0f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae1d0f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a6141d700 10 monclient: _renew_subs 2023-09-30T01:54:48.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a6141d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f8a6141d700 1 -- 172.21.15.7:0/3745188111 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a5c06e2d0 con 0x7f8a5c096a50 2023-09-30T01:54:48.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae1d0f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae1d0f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae1d0f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f08817de700 1 -- 172.21.15.7:0/4220366267 shutdown_connections 2023-09-30T01:54:48.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f08817de700 1 --2- 172.21.15.7:0/4220366267 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f087c06f510 0x7f087c09b010 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f08817de700 1 --2- 172.21.15.7:0/4220366267 >> v2:172.21.15.7:3300/0 conn(0x7f087c069b70 0x7f087c09dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f08817de700 1 -- 172.21.15.7:0/4220366267 >> 172.21.15.7:0/4220366267 conn(0x7f087c0a32a0 msgr2=0x7f087c0a36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f4457da8700 10 monclient: shutdown 2023-09-30T01:54:48.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f4457da8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f4457da8700 1 -- 172.21.15.7:0/2117379672 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f445009bcb0 msgr2=0x7f445009ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f4457da8700 1 --2- 172.21.15.7:0/2117379672 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f445009bcb0 0x7f445009ce30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f44400098e0 tx=0x7f44400091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.212+0000 7f4457da8700 1 -- 172.21.15.7:0/2117379672 shutdown_connections 2023-09-30T01:54:48.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.212+0000 7f4457da8700 1 --2- 172.21.15.7:0/2117379672 >> v2:172.21.15.7:3300/0 conn(0x7f445009c520 0x7f445009c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae2510700 1 --2- 172.21.15.7:0/3161612665 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6adc06afc0 0x7f6adc141da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae2510700 10 monclient(hunting): get_auth_request con 0x7f6adc06afc0 auth_method 0 2023-09-30T01:54:48.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae2510700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae2510700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae2510700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae2d11700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae2d11700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae2d11700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.211+0000 7f6ae1d0f700 10 monclient(hunting): handle_auth_done global_id 4244 payload 824 2023-09-30T01:54:48.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.212+0000 7f6ae1d0f700 1 -- 172.21.15.7:0/3161612665 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6adc06afc0 msgr2=0x7f6adc141da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.212+0000 7f6ae1d0f700 1 --2- 172.21.15.7:0/3161612665 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6adc06afc0 0x7f6adc141da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.212+0000 7f6ae1d0f700 1 -- 172.21.15.7:0/3161612665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6adc132df0 msgr2=0x7f6adc0986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.212+0000 7f6ae1d0f700 1 --2- 172.21.15.7:0/3161612665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6adc132df0 0x7f6adc0986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.212+0000 7f6ae1d0f700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.212+0000 7f6ae1d0f700 1 monclient: found mon.noname-c 2023-09-30T01:54:48.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.212+0000 7f6ae1d0f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.212+0000 7f6ae1d0f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.279 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.212+0000 7f6ae1d0f700 1 -- 172.21.15.7:0/3161612665 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad4009a20 con 0x7f6adc132310 2023-09-30T01:54:48.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.212+0000 7f6ae2d11700 1 --2- 172.21.15.7:0/3161612665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6adc132df0 0x7f6adc0986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:48.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.212+0000 7f4457da8700 1 --2- 172.21.15.7:0/2117379672 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f445009bcb0 0x7f445009ce30 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f44400098e0 tx=0x7f44400091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.212+0000 7f4457da8700 1 -- 172.21.15.7:0/2117379672 >> 172.21.15.7:0/2117379672 conn(0x7f44500a1b30 msgr2=0x7f44500a1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.212+0000 7f4457da8700 1 -- 172.21.15.7:0/2117379672 shutdown_connections 2023-09-30T01:54:48.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.212+0000 7f08817de700 1 -- 172.21.15.7:0/4220366267 shutdown_connections 2023-09-30T01:54:48.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.212+0000 7f08817de700 1 -- 172.21.15.7:0/4220366267 wait complete. 2023-09-30T01:54:48.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.212+0000 7f8a6141d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.212+0000 7f8a6141d700 1 -- 172.21.15.7:0/3745188111 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8a5c12f560 con 0x7f8a5c096a50 2023-09-30T01:54:48.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.212+0000 7f4457da8700 1 -- 172.21.15.7:0/2117379672 wait complete. 2023-09-30T01:54:48.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.212+0000 7f8a42ffd700 1 -- 172.21.15.7:0/3745188111 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f8a5001aaa0 con 0x7f8a5c096a50 2023-09-30T01:54:48.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.213+0000 7f08817de700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.213+0000 7f08817de700 10 monclient: monmap: 2023-09-30T01:54:48.282 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.282 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:48.282 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:48.282 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:48.282 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.283 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.283 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:48.283 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:48.283 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:48.283 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.213+0000 7f8a42ffd700 1 --2- 172.21.15.7:0/3745188111 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f8a48035830 0x7f8a48037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.213+0000 7f8a42ffd700 1 -- 172.21.15.7:0/3745188111 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f8a500497d0 con 0x7f8a5c096a50 2023-09-30T01:54:48.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.213+0000 7f8a42ffd700 1 --2- 172.21.15.7:0/3745188111 >> [v2:172.21.15.50:6803/86632,v1:172.21.15.50:6806/86632] conn(0x7f8a4803b730 0x7f8a4803db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.213+0000 7f8a42ffd700 1 -- 172.21.15.7:0/3745188111 --> [v2:172.21.15.50:6803/86632,v1:172.21.15.50:6806/86632] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8a4803e1f0 con 0x7f8a4803b730 2023-09-30T01:54:48.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.213+0000 7f8a42ffd700 1 -- 172.21.15.7:0/3745188111 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a5004bd60 con 0x7f8a5c096a50 2023-09-30T01:54:48.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.213+0000 7f8a42ffd700 10 monclient: handle_get_version_reply finishing 1 version 15 2023-09-30T01:54:48.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.213+0000 7f08817de700 1 Processor -- start 2023-09-30T01:54:48.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.213+0000 7f4457da8700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.213+0000 7f4457da8700 10 monclient: monmap: 2023-09-30T01:54:48.285 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.285 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:48.285 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:48.285 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:48.285 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.285 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.286 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:48.286 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:48.286 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:48.286 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.213+0000 7f4457da8700 1 Processor -- start 2023-09-30T01:54:48.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.213+0000 7f4457da8700 1 -- start start 2023-09-30T01:54:48.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.213+0000 7f4457da8700 10 monclient: init 2023-09-30T01:54:48.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.213+0000 7f8a5b7fe700 1 --2- 172.21.15.7:0/3745188111 >> [v2:172.21.15.50:6803/86632,v1:172.21.15.50:6806/86632] conn(0x7f8a4803b730 0x7f8a4803db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.213+0000 7f8a5a7fc700 1 --2- 172.21.15.7:0/3745188111 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f8a48035830 0x7f8a48037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.213+0000 7f8a5b7fe700 10 monclient: get_auth_request con 0x7f8a4803b730 auth_method 0 2023-09-30T01:54:48.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.213+0000 7f8a5a7fc700 10 monclient: get_auth_request con 0x7f8a48035830 auth_method 0 2023-09-30T01:54:48.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.213+0000 7f08817de700 1 -- start start 2023-09-30T01:54:48.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.213+0000 7f08817de700 10 monclient: init 2023-09-30T01:54:48.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f08817de700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4457da8700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4457da8700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:54:48.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f08817de700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:54:48.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f08817de700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f087c069b70 0x7f087c13e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f08817de700 10 monclient(hunting): picked mon.noname-c con 0x7f087c069b70 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:48.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f08817de700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f087c06f510 0x7f087c13d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f08817de700 10 monclient(hunting): picked mon.noname-a con 0x7f087c06f510 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:48.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f08817de700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f087c06f8e0 0x7f087c141770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f08817de700 10 monclient(hunting): picked mon.noname-b con 0x7f087c06f8e0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:48.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f08817de700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4457da8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f445009c520 0x7f445013a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4457da8700 10 monclient(hunting): picked mon.noname-a con 0x7f445009c520 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:48.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4457da8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f445013ab20 0x7f44501350f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f08817de700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c122f10 con 0x7f087c06f510 2023-09-30T01:54:48.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f08817de700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f08817de700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c13ed40 con 0x7f087c06f8e0 2023-09-30T01:54:48.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f08817de700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f08817de700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087c120eb0 con 0x7f087c069b70 2023-09-30T01:54:48.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f08817de700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f08817de700 10 monclient(hunting): authenticate will time out at 1177.279336s 2023-09-30T01:54:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4457da8700 10 monclient(hunting): picked mon.noname-b con 0x7f445013ab20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4457da8700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4450119070 0x7f4450138da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4457da8700 10 monclient(hunting): picked mon.noname-c con 0x7f4450119070 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4457da8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4457da8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445009b640 con 0x7f445009c520 2023-09-30T01:54:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4457da8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4457da8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445006c5f0 con 0x7f445013ab20 2023-09-30T01:54:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4457da8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4457da8700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f445011a240 con 0x7f4450119070 2023-09-30T01:54:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4457da8700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4457da8700 10 monclient(hunting): authenticate will time out at 1177.279339s 2023-09-30T01:54:48.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f087bfff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f087c069b70 0x7f087c13e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f6ae1d0f700 1 -- 172.21.15.7:0/3161612665 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6adc132310 msgr2=0x7f6adc12f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f6ae1d0f700 1 --2- 172.21.15.7:0/3161612665 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6adc132310 0x7f6adc12f5a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6acc00c530 tx=0x7f6acc007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f087bfff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f087c069b70 0x7f087c13e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59386/0 (socket says 172.21.15.7:59386) 2023-09-30T01:54:48.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f087bfff700 1 -- 172.21.15.7:0/1387115715 learned_addr learned my addr 172.21.15.7:0/1387115715 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:48.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f087bfff700 10 monclient(hunting): get_auth_request con 0x7f087c069b70 auth_method 0 2023-09-30T01:54:48.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f087bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f087bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f087bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4455b44700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f445009c520 0x7f445013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4455b44700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f445009c520 0x7f445013a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54742/0 (socket says 172.21.15.7:54742) 2023-09-30T01:54:48.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4455b44700 1 -- 172.21.15.7:0/1167555467 learned_addr learned my addr 172.21.15.7:0/1167555467 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:48.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4455b44700 10 monclient(hunting): get_auth_request con 0x7f445009c520 auth_method 0 2023-09-30T01:54:48.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4455b44700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4455b44700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4455b44700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4455343700 1 --2- 172.21.15.7:0/1167555467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f445013ab20 0x7f44501350f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4455343700 10 monclient(hunting): get_auth_request con 0x7f445013ab20 auth_method 0 2023-09-30T01:54:48.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4455343700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4455343700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4455343700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4456345700 1 --2- 172.21.15.7:0/1167555467 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4450119070 0x7f4450138da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4456345700 10 monclient(hunting): get_auth_request con 0x7f4450119070 auth_method 0 2023-09-30T01:54:48.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4456345700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4456345700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4456345700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f0880fdd700 1 --2- 172.21.15.7:0/1387115715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f087c06f8e0 0x7f087c141770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f087b7fe700 1 --2- 172.21.15.7:0/1387115715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f087c06f510 0x7f087c13d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f0880fdd700 10 monclient(hunting): get_auth_request con 0x7f087c06f8e0 auth_method 0 2023-09-30T01:54:48.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f0880fdd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f0880fdd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f0880fdd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f087b7fe700 10 monclient(hunting): get_auth_request con 0x7f087c06f510 auth_method 0 2023-09-30T01:54:48.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f087b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f087b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f087b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f087bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f087bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f087bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f6ad27fc700 1 -- 172.21.15.7:0/3161612665 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6acc005d40 con 0x7f6adc132310 2023-09-30T01:54:48.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f6ad27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f6ad27fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:48.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f0880fdd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f0880fdd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f0880fdd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4455343700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4455343700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4455343700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4456345700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4456345700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4456345700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f6ad27fc700 10 monclient: dump: 2023-09-30T01:54:48.299 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.299 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.299 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.299 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.299 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.299 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.299 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.299 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.299 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.299 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f6ad27fc700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f6ad27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f6ae4774700 5 monclient: authenticate success, global_id 4244 2023-09-30T01:54:48.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f6ae4774700 10 monclient: _renew_subs 2023-09-30T01:54:48.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f6ae4774700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f6ae4774700 1 -- 172.21.15.7:0/3161612665 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6adc11e800 con 0x7f6adc132310 2023-09-30T01:54:48.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4455b44700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4455b44700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f4455b44700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f6ae4774700 10 monclient: _renew_subs 2023-09-30T01:54:48.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f6ae4774700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.214+0000 7f6ae4774700 1 -- 172.21.15.7:0/3161612665 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6adc09a260 con 0x7f6adc132310 2023-09-30T01:54:48.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f087b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f087b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f087b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f4455343700 10 monclient(hunting): handle_auth_done global_id 4246 payload 824 2023-09-30T01:54:48.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f4455343700 1 -- 172.21.15.7:0/1167555467 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4450119070 msgr2=0x7f4450138da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f4455343700 1 --2- 172.21.15.7:0/1167555467 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4450119070 0x7f4450138da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f4455343700 1 -- 172.21.15.7:0/1167555467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f445009c520 msgr2=0x7f445013a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f4455343700 1 --2- 172.21.15.7:0/1167555467 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f445009c520 0x7f445013a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f4455343700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f4455343700 1 monclient: found mon.noname-b 2023-09-30T01:54:48.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f4455343700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f4455343700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3300/0 2023-09-30T01:54:48.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f4455343700 1 -- 172.21.15.7:0/1167555467 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4440009420 con 0x7f445013ab20 2023-09-30T01:54:48.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f087bfff700 10 monclient(hunting): handle_auth_done global_id 4247 payload 824 2023-09-30T01:54:48.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f087bfff700 1 -- 172.21.15.7:0/1387115715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f087c06f8e0 msgr2=0x7f087c141770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f087bfff700 1 --2- 172.21.15.7:0/1387115715 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f087c06f8e0 0x7f087c141770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f087bfff700 1 -- 172.21.15.7:0/1387115715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f087c06f510 msgr2=0x7f087c13d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f087bfff700 1 --2- 172.21.15.7:0/1387115715 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f087c06f510 0x7f087c13d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f087bfff700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f087bfff700 1 monclient: found mon.noname-c 2023-09-30T01:54:48.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f087bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f087bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f087bfff700 1 -- 172.21.15.7:0/1387115715 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c0094f0 con 0x7f087c069b70 2023-09-30T01:54:48.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f6ad27fc700 1 -- 172.21.15.7:0/3161612665 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6acc0070d0 con 0x7f6adc132310 2023-09-30T01:54:48.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f6ad27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f6ad27fc700 1 -- 172.21.15.7:0/3161612665 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6acc005ea0 con 0x7f6adc132310 2023-09-30T01:54:48.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f6ad27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f6ad27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:48.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f6ad27fc700 10 monclient: dump: 2023-09-30T01:54:48.305 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.306 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.306 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.306 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.306 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.306 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.306 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.306 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.306 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.306 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f6ae0d0d700 4 set_mon_vals no callback set 2023-09-30T01:54:48.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f6ae4774700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.215+0000 7f6ae4774700 1 -- 172.21.15.7:0/3161612665 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6adc09e7c0 con 0x7f6adc132310 2023-09-30T01:54:48.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.216+0000 7f6ad27fc700 1 -- 172.21.15.7:0/3161612665 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f6acc0076a0 con 0x7f6adc132310 2023-09-30T01:54:48.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.216+0000 7f6ad27fc700 1 --2- 172.21.15.7:0/3161612665 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6ac00357e0 0x7f6ac0037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.216+0000 7f6ad27fc700 1 -- 172.21.15.7:0/3161612665 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f6acc0471d0 con 0x7f6adc132310 2023-09-30T01:54:48.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.216+0000 7f6ad27fc700 1 --2- 172.21.15.7:0/3161612665 >> [v2:172.21.15.50:6800/86634,v1:172.21.15.50:6801/86634] conn(0x7f6ac003b5e0 0x7f6ac003d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.216+0000 7f6ad27fc700 1 -- 172.21.15.7:0/3161612665 --> [v2:172.21.15.50:6800/86634,v1:172.21.15.50:6801/86634] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ac003e0a0 con 0x7f6ac003b5e0 2023-09-30T01:54:48.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.216+0000 7f6ad27fc700 1 -- 172.21.15.7:0/3161612665 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6acc0497a0 con 0x7f6adc132310 2023-09-30T01:54:48.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.216+0000 7f6ad27fc700 10 monclient: handle_get_version_reply finishing 1 version 15 2023-09-30T01:54:48.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.216+0000 7f6ae2510700 1 --2- 172.21.15.7:0/3161612665 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6ac00357e0 0x7f6ac0037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.216+0000 7f6ae2510700 10 monclient: get_auth_request con 0x7f6ac00357e0 auth_method 0 2023-09-30T01:54:48.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.216+0000 7f8a5a7fc700 1 --2- 172.21.15.7:0/3745188111 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f8a48035830 0x7f8a48037ce0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c006fd0 tx=0x7f8a4c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.216+0000 7f6ae2d11700 1 --2- 172.21.15.7:0/3161612665 >> [v2:172.21.15.50:6800/86634,v1:172.21.15.50:6801/86634] conn(0x7f6ac003b5e0 0x7f6ac003d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.216+0000 7f6ae2d11700 10 monclient: get_auth_request con 0x7f6ac003b5e0 auth_method 0 2023-09-30T01:54:48.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4455343700 1 -- 172.21.15.7:0/1167555467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f445013ab20 msgr2=0x7f44501350f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4455343700 1 --2- 172.21.15.7:0/1167555467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f445013ab20 0x7f44501350f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4450068c90 tx=0x7f4440016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f087bfff700 1 -- 172.21.15.7:0/1387115715 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f087c069b70 msgr2=0x7f087c13e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f087bfff700 1 --2- 172.21.15.7:0/1387115715 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f087c069b70 0x7f087c13e200 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f086c003080 tx=0x7f086c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4445ffb700 1 -- 172.21.15.7:0/1167555467 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444000bb10 con 0x7f445013ab20 2023-09-30T01:54:48.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4445ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4445ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:54:48.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4445ffb700 10 monclient: dump: 2023-09-30T01:54:48.310 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.310 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.310 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.310 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.310 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.310 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.310 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.311 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.311 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.311 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4445ffb700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4445ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4457da8700 5 monclient: authenticate success, global_id 4246 2023-09-30T01:54:48.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4445ffb700 1 -- 172.21.15.7:0/1167555467 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4440020070 con 0x7f445013ab20 2023-09-30T01:54:48.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4445ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4445ffb700 1 -- 172.21.15.7:0/1167555467 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444001b450 con 0x7f445013ab20 2023-09-30T01:54:48.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4445ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4445ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:54:48.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4445ffb700 10 monclient: dump: 2023-09-30T01:54:48.311 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.312 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4457da8700 10 monclient: _renew_subs 2023-09-30T01:54:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4447fff700 4 set_mon_vals no callback set 2023-09-30T01:54:48.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4457da8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4457da8700 1 -- 172.21.15.7:0/1167555467 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f445013f0d0 con 0x7f445013ab20 2023-09-30T01:54:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f0863fff700 1 -- 172.21.15.7:0/1387115715 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c006b30 con 0x7f087c069b70 2023-09-30T01:54:48.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f0863fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f0863fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:48.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f0863fff700 10 monclient: dump: 2023-09-30T01:54:48.314 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.314 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.314 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.314 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.314 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.314 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.314 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.314 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.315 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.315 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f0863fff700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f0863fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f0863fff700 1 -- 172.21.15.7:0/1387115715 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086c020070 con 0x7f087c069b70 2023-09-30T01:54:48.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f08817de700 5 monclient: authenticate success, global_id 4247 2023-09-30T01:54:48.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f0863fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f08817de700 10 monclient: _renew_subs 2023-09-30T01:54:48.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f0863fff700 1 -- 172.21.15.7:0/1387115715 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c000c80 con 0x7f087c069b70 2023-09-30T01:54:48.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f08817de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f08817de700 1 -- 172.21.15.7:0/1387115715 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087c122950 con 0x7f087c069b70 2023-09-30T01:54:48.315 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f0863fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f087a7fc700 4 set_mon_vals no callback set 2023-09-30T01:54:48.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f0863fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:48.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f0863fff700 10 monclient: dump: 2023-09-30T01:54:48.316 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.316 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.316 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.316 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.316 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.316 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.316 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.316 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.316 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.316 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4457da8700 10 monclient: _renew_subs 2023-09-30T01:54:48.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4457da8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:48.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.217+0000 7f4457da8700 1 -- 172.21.15.7:0/1167555467 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44501392e0 con 0x7f445013ab20 2023-09-30T01:54:48.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.218+0000 7f08817de700 10 monclient: _renew_subs 2023-09-30T01:54:48.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.218+0000 7f08817de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.218+0000 7f08817de700 1 -- 172.21.15.7:0/1387115715 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f087c13ddf0 con 0x7f087c069b70 2023-09-30T01:54:48.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.218+0000 7f8a5b7fe700 1 --2- 172.21.15.7:0/3745188111 >> [v2:172.21.15.50:6803/86632,v1:172.21.15.50:6806/86632] conn(0x7f8a4803b730 0x7f8a4803db40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.318 INFO:teuthology.orchestra.run.smithi007.stdout:47244640259 2023-09-30T01:54:48.319 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T01:54:48.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.218+0000 7f4457da8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:48.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.218+0000 7f4457da8700 1 -- 172.21.15.7:0/1167555467 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4450135630 con 0x7f445013ab20 2023-09-30T01:54:48.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.218+0000 7f6ae2510700 1 --2- 172.21.15.7:0/3161612665 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6ac00357e0 0x7f6ac0037c90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6ad80099d0 tx=0x7f6ad8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.218+0000 7f0863fff700 1 -- 172.21.15.7:0/1387115715 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f086c022030 con 0x7f087c069b70 2023-09-30T01:54:48.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.219+0000 7f4445ffb700 1 -- 172.21.15.7:0/1167555467 <== mon.1 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f4440022030 con 0x7f445013ab20 2023-09-30T01:54:48.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.219+0000 7f0863fff700 1 --2- 172.21.15.7:0/1387115715 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f08680357e0 0x7f0868037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.219+0000 7f0863fff700 1 -- 172.21.15.7:0/1387115715 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f086c049870 con 0x7f087c069b70 2023-09-30T01:54:48.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.219+0000 7f4445ffb700 1 --2- 172.21.15.7:0/1167555467 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f4434035830 0x7f4434037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.219+0000 7f8a42ffd700 1 -- 172.21.15.7:0/3745188111 <== osd.4 v2:172.21.15.50:6803/86632 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f8a4803e1f0 con 0x7f8a4803b730 2023-09-30T01:54:48.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.219+0000 7f4445ffb700 1 -- 172.21.15.7:0/1167555467 <== mon.1 v2:172.21.15.50:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f44400498e0 con 0x7f445013ab20 2023-09-30T01:54:48.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.219+0000 7f087b7fe700 1 --2- 172.21.15.7:0/1387115715 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f08680357e0 0x7f0868037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.219+0000 7f4445ffb700 1 --2- 172.21.15.7:0/1167555467 >> [v2:172.21.15.7:6808/86602,v1:172.21.15.7:6809/86602] conn(0x7f443403b730 0x7f443403db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.219+0000 7f087b7fe700 10 monclient: get_auth_request con 0x7f08680357e0 auth_method 0 2023-09-30T01:54:48.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.219+0000 7f4445ffb700 1 -- 172.21.15.7:0/1167555467 --> [v2:172.21.15.7:6808/86602,v1:172.21.15.7:6809/86602] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f443403e1f0 con 0x7f443403b730 2023-09-30T01:54:48.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.219+0000 7f4445ffb700 1 -- 172.21.15.7:0/1167555467 <== mon.1 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f444004e2a0 con 0x7f445013ab20 2023-09-30T01:54:48.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.219+0000 7f4445ffb700 10 monclient: handle_get_version_reply finishing 1 version 15 2023-09-30T01:54:48.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.219+0000 7f4455b44700 1 --2- 172.21.15.7:0/1167555467 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f4434035830 0x7f4434037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.219+0000 7f4455b44700 10 monclient: get_auth_request con 0x7f4434035830 auth_method 0 2023-09-30T01:54:48.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.219+0000 7f4456345700 1 --2- 172.21.15.7:0/1167555467 >> [v2:172.21.15.7:6808/86602,v1:172.21.15.7:6809/86602] conn(0x7f443403b730 0x7f443403db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.219+0000 7f08817de700 1 --2- 172.21.15.7:0/1387115715 >> [v2:172.21.15.7:6800/86601,v1:172.21.15.7:6801/86601] conn(0x7f087c121a80 0x7f087c1421c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.219+0000 7f4456345700 10 monclient: get_auth_request con 0x7f443403b730 auth_method 0 2023-09-30T01:54:48.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.219+0000 7f08817de700 1 -- 172.21.15.7:0/1387115715 --> [v2:172.21.15.7:6800/86601,v1:172.21.15.7:6801/86601] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f087c09e9e0 con 0x7f087c121a80 2023-09-30T01:54:48.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.219+0000 7f0880fdd700 1 --2- 172.21.15.7:0/1387115715 >> [v2:172.21.15.7:6800/86601,v1:172.21.15.7:6801/86601] conn(0x7f087c121a80 0x7f087c1421c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.219+0000 7f0880fdd700 10 monclient: get_auth_request con 0x7f087c121a80 auth_method 0 2023-09-30T01:54:48.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.221+0000 7f6ae2d11700 1 --2- 172.21.15.7:0/3161612665 >> [v2:172.21.15.50:6800/86634,v1:172.21.15.50:6801/86634] conn(0x7f6ac003b5e0 0x7f6ac003d9f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.221+0000 7f6ad27fc700 1 -- 172.21.15.7:0/3161612665 <== osd.5 v2:172.21.15.50:6800/86634 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f6ac003e0a0 con 0x7f6ac003b5e0 2023-09-30T01:54:48.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.221+0000 7f087b7fe700 1 --2- 172.21.15.7:0/1387115715 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f08680357e0 0x7f0868037c90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f087c069f80 tx=0x7f08700091a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.222+0000 7f4455b44700 1 --2- 172.21.15.7:0/1167555467 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f4434035830 0x7f4434037ce0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f444800f630 tx=0x7f444800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.224+0000 7f4456345700 1 --2- 172.21.15.7:0/1167555467 >> [v2:172.21.15.7:6808/86602,v1:172.21.15.7:6809/86602] conn(0x7f443403b730 0x7f443403db40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.224+0000 7f0880fdd700 1 --2- 172.21.15.7:0/1387115715 >> [v2:172.21.15.7:6800/86601,v1:172.21.15.7:6801/86601] conn(0x7f087c121a80 0x7f087c1421c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.224+0000 7f4445ffb700 1 -- 172.21.15.7:0/1167555467 <== osd.2 v2:172.21.15.7:6808/86602 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f443403e1f0 con 0x7f443403b730 2023-09-30T01:54:48.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.224+0000 7f0863fff700 1 -- 172.21.15.7:0/1387115715 <== osd.0 v2:172.21.15.7:6800/86601 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f087c09e9e0 con 0x7f087c121a80 2023-09-30T01:54:48.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.234+0000 7f8a6141d700 1 -- 172.21.15.7:0/3745188111 --> [v2:172.21.15.50:6803/86632,v1:172.21.15.50:6806/86632] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8a5c064c10 con 0x7f8a4803b730 2023-09-30T01:54:48.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.235+0000 7f8a42ffd700 1 -- 172.21.15.7:0/3745188111 <== osd.4 v2:172.21.15.50:6803/86632 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8a5c064c10 con 0x7f8a4803b730 2023-09-30T01:54:48.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.235+0000 7f8a6141d700 1 -- 172.21.15.7:0/3745188111 >> [v2:172.21.15.50:6803/86632,v1:172.21.15.50:6806/86632] conn(0x7f8a4803b730 msgr2=0x7f8a4803db40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.235+0000 7f8a6141d700 1 --2- 172.21.15.7:0/3745188111 >> [v2:172.21.15.50:6803/86632,v1:172.21.15.50:6806/86632] conn(0x7f8a4803b730 0x7f8a4803db40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.235+0000 7f8a6141d700 1 -- 172.21.15.7:0/3745188111 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f8a48035830 msgr2=0x7f8a48037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.235+0000 7f8a6141d700 1 --2- 172.21.15.7:0/3745188111 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f8a48035830 0x7f8a48037ce0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c006fd0 tx=0x7f8a4c008040 comp rx=0 tx=0).stop 2023-09-30T01:54:48.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.235+0000 7f8a6141d700 10 monclient: shutdown 2023-09-30T01:54:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.235+0000 7f8a6141d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.235+0000 7f8a6141d700 1 -- 172.21.15.7:0/3745188111 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8a5c096a50 msgr2=0x7f8a5c06e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.235+0000 7f8a6141d700 1 --2- 172.21.15.7:0/3745188111 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8a5c096a50 0x7f8a5c06e9e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8a5000b0e0 tx=0x7f8a50016000 comp rx=0 tx=0).stop 2023-09-30T01:54:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.235+0000 7f8a5b7fe700 1 -- 172.21.15.7:0/3745188111 reap_dead start 2023-09-30T01:54:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.235+0000 7f8a6141d700 1 -- 172.21.15.7:0/3745188111 shutdown_connections 2023-09-30T01:54:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.235+0000 7f8a6141d700 1 -- 172.21.15.7:0/3745188111 >> 172.21.15.7:0/3745188111 conn(0x7f8a5c070cf0 msgr2=0x7f8a5c135920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.235+0000 7f8a6141d700 1 -- 172.21.15.7:0/3745188111 shutdown_connections 2023-09-30T01:54:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.235+0000 7f8a6141d700 1 -- 172.21.15.7:0/3745188111 wait complete. 2023-09-30T01:54:48.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.240+0000 7f08817de700 1 -- 172.21.15.7:0/1387115715 --> [v2:172.21.15.7:6800/86601,v1:172.21.15.7:6801/86601] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f087c1206e0 con 0x7f087c121a80 2023-09-30T01:54:48.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.240+0000 7f0863fff700 1 -- 172.21.15.7:0/1387115715 <== osd.0 v2:172.21.15.7:6800/86601 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f087c1206e0 con 0x7f087c121a80 2023-09-30T01:54:48.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.240+0000 7f08817de700 1 -- 172.21.15.7:0/1387115715 >> [v2:172.21.15.7:6800/86601,v1:172.21.15.7:6801/86601] conn(0x7f087c121a80 msgr2=0x7f087c1421c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.240+0000 7f08817de700 1 --2- 172.21.15.7:0/1387115715 >> [v2:172.21.15.7:6800/86601,v1:172.21.15.7:6801/86601] conn(0x7f087c121a80 0x7f087c1421c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.241+0000 7f08817de700 1 -- 172.21.15.7:0/1387115715 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f08680357e0 msgr2=0x7f0868037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.241+0000 7f08817de700 1 --2- 172.21.15.7:0/1387115715 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f08680357e0 0x7f0868037c90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f087c069f80 tx=0x7f08700091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.241+0000 7f08817de700 10 monclient: shutdown 2023-09-30T01:54:48.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.241+0000 7f08817de700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.241+0000 7f08817de700 1 -- 172.21.15.7:0/1387115715 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f087c069b70 msgr2=0x7f087c13e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.241+0000 7f08817de700 1 --2- 172.21.15.7:0/1387115715 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f087c069b70 0x7f087c13e200 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f086c003080 tx=0x7f086c016000 comp rx=0 tx=0).stop 2023-09-30T01:54:48.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.241+0000 7f0880fdd700 1 -- 172.21.15.7:0/1387115715 reap_dead start 2023-09-30T01:54:48.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.241+0000 7f08817de700 1 -- 172.21.15.7:0/1387115715 shutdown_connections 2023-09-30T01:54:48.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.241+0000 7f08817de700 1 -- 172.21.15.7:0/1387115715 >> 172.21.15.7:0/1387115715 conn(0x7f087c0a32a0 msgr2=0x7f087c09d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.241+0000 7f08817de700 1 -- 172.21.15.7:0/1387115715 shutdown_connections 2023-09-30T01:54:48.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.241+0000 7f08817de700 1 -- 172.21.15.7:0/1387115715 wait complete. 2023-09-30T01:54:48.333 INFO:teuthology.orchestra.run.smithi007.stdout:47244640258 2023-09-30T01:54:48.333 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T01:54:48.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.250+0000 7f6ae4774700 1 -- 172.21.15.7:0/3161612665 --> [v2:172.21.15.50:6800/86634,v1:172.21.15.50:6801/86634] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6adc142580 con 0x7f6ac003b5e0 2023-09-30T01:54:48.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.251+0000 7f6ad27fc700 1 -- 172.21.15.7:0/3161612665 <== osd.5 v2:172.21.15.50:6800/86634 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6adc142580 con 0x7f6ac003b5e0 2023-09-30T01:54:48.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.251+0000 7f6ae4774700 1 -- 172.21.15.7:0/3161612665 >> [v2:172.21.15.50:6800/86634,v1:172.21.15.50:6801/86634] conn(0x7f6ac003b5e0 msgr2=0x7f6ac003d9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.251+0000 7f6ae4774700 1 --2- 172.21.15.7:0/3161612665 >> [v2:172.21.15.50:6800/86634,v1:172.21.15.50:6801/86634] conn(0x7f6ac003b5e0 0x7f6ac003d9f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.252+0000 7f6ae4774700 1 -- 172.21.15.7:0/3161612665 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6ac00357e0 msgr2=0x7f6ac0037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.252+0000 7f6ae4774700 1 --2- 172.21.15.7:0/3161612665 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6ac00357e0 0x7f6ac0037c90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6ad80099d0 tx=0x7f6ad8008040 comp rx=0 tx=0).stop 2023-09-30T01:54:48.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.252+0000 7f6ae4774700 10 monclient: shutdown 2023-09-30T01:54:48.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.252+0000 7f6ae4774700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.252+0000 7f6ae4774700 1 -- 172.21.15.7:0/3161612665 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6adc132310 msgr2=0x7f6adc12f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.252+0000 7f6ae4774700 1 --2- 172.21.15.7:0/3161612665 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6adc132310 0x7f6adc12f5a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6acc00c530 tx=0x7f6acc007f90 comp rx=0 tx=0).stop 2023-09-30T01:54:48.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.252+0000 7f6ae2d11700 1 -- 172.21.15.7:0/3161612665 reap_dead start 2023-09-30T01:54:48.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.252+0000 7f6ae4774700 1 -- 172.21.15.7:0/3161612665 shutdown_connections 2023-09-30T01:54:48.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.252+0000 7f6ae4774700 1 -- 172.21.15.7:0/3161612665 >> 172.21.15.7:0/3161612665 conn(0x7f6adc067f30 msgr2=0x7f6adc06e760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.252+0000 7f6ae4774700 1 -- 172.21.15.7:0/3161612665 shutdown_connections 2023-09-30T01:54:48.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.252+0000 7f6ae4774700 1 -- 172.21.15.7:0/3161612665 wait complete. 2023-09-30T01:54:48.336 INFO:teuthology.orchestra.run.smithi007.stdout:47244640258 2023-09-30T01:54:48.337 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T01:54:48.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.254+0000 7f4457da8700 1 -- 172.21.15.7:0/1167555467 --> [v2:172.21.15.7:6808/86602,v1:172.21.15.7:6809/86602] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f44500a0280 con 0x7f443403b730 2023-09-30T01:54:48.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.254+0000 7f4445ffb700 1 -- 172.21.15.7:0/1167555467 <== osd.2 v2:172.21.15.7:6808/86602 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f44500a0280 con 0x7f443403b730 2023-09-30T01:54:48.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.255+0000 7f4457da8700 1 -- 172.21.15.7:0/1167555467 >> [v2:172.21.15.7:6808/86602,v1:172.21.15.7:6809/86602] conn(0x7f443403b730 msgr2=0x7f443403db40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.255+0000 7f4457da8700 1 --2- 172.21.15.7:0/1167555467 >> [v2:172.21.15.7:6808/86602,v1:172.21.15.7:6809/86602] conn(0x7f443403b730 0x7f443403db40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.255+0000 7f4457da8700 1 -- 172.21.15.7:0/1167555467 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f4434035830 msgr2=0x7f4434037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.255+0000 7f4457da8700 1 --2- 172.21.15.7:0/1167555467 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f4434035830 0x7f4434037ce0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f444800f630 tx=0x7f444800b040 comp rx=0 tx=0).stop 2023-09-30T01:54:48.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.255+0000 7f4457da8700 10 monclient: shutdown 2023-09-30T01:54:48.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.255+0000 7f4457da8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.255+0000 7f4457da8700 1 -- 172.21.15.7:0/1167555467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f445013ab20 msgr2=0x7f44501350f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.255+0000 7f4457da8700 1 --2- 172.21.15.7:0/1167555467 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f445013ab20 0x7f44501350f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4450068c90 tx=0x7f4440016000 comp rx=0 tx=0).stop 2023-09-30T01:54:48.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.255+0000 7f4456345700 1 -- 172.21.15.7:0/1167555467 reap_dead start 2023-09-30T01:54:48.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.255+0000 7f4457da8700 1 -- 172.21.15.7:0/1167555467 shutdown_connections 2023-09-30T01:54:48.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.255+0000 7f4457da8700 1 -- 172.21.15.7:0/1167555467 >> 172.21.15.7:0/1167555467 conn(0x7f44500a1b30 msgr2=0x7f445009f930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.255+0000 7f4457da8700 1 -- 172.21.15.7:0/1167555467 shutdown_connections 2023-09-30T01:54:48.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.255+0000 7f4457da8700 1 -- 172.21.15.7:0/1167555467 wait complete. 2023-09-30T01:54:48.340 INFO:teuthology.orchestra.run.smithi007.stdout:47244640259 2023-09-30T01:54:48.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T01:54:48.341 INFO:teuthology.orchestra.run.smithi007.stdout:47244640259 2023-09-30T01:54:48.341 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T01:54:48.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.491+0000 7ffa7b08f700 10 monclient: get_monmap_and_config 2023-09-30T01:54:48.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.491+0000 7ffa7b08f700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.492 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.491+0000 7ffa7b08f700 10 monclient: monmap: 2023-09-30T01:54:48.492 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.492 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.492 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:48.492989+0000 2023-09-30T01:54:48.492 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:48.492989+0000 2023-09-30T01:54:48.493 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.493 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.493 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:48.493 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:48.493 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:48.493 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:48.493 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:48.493 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.492+0000 7ffa7b08f700 1 Processor -- start 2023-09-30T01:54:48.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.492+0000 7ffa7b08f700 1 -- start start 2023-09-30T01:54:48.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.492+0000 7ffa7b08f700 10 monclient: init 2023-09-30T01:54:48.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.493+0000 7ffa7b08f700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.493+0000 7ffa7b08f700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-09-30T01:54:48.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.493+0000 7ffa7b08f700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa74096a50 0x7ffa74132680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.493+0000 7ffa7b08f700 10 monclient(hunting): picked mon.noname-b con 0x7ffa74096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:48.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.493+0000 7ffa7b08f700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ffa74096e20 0x7ffa740971f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.493+0000 7ffa7b08f700 10 monclient(hunting): picked mon.noname-c con 0x7ffa74096e20 addr v2:172.21.15.50:3300/0 2023-09-30T01:54:48.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.493+0000 7ffa7b08f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa7412ef00 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:48.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.493+0000 7ffa7b08f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa74099440 con 0x7ffa7412ef00 2023-09-30T01:54:48.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.493+0000 7ffa7b08f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.493+0000 7ffa7b08f700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa74097b40 con 0x7ffa74096e20 2023-09-30T01:54:48.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.493+0000 7ffa7b08f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.493+0000 7ffa7b08f700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa74099160 con 0x7ffa74096a50 2023-09-30T01:54:48.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.493+0000 7ffa7b08f700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.493+0000 7ffa7b08f700 10 monclient(hunting): authenticate will time out at 1177.558475s 2023-09-30T01:54:48.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.493+0000 7ffa7962c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ffa7412ef00 0x7ffa7412f370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58092/0 (socket says 172.21.15.7:58092) 2023-09-30T01:54:48.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.493+0000 7ffa7962c700 1 -- 172.21.15.7:0/1629127331 learned_addr learned my addr 172.21.15.7:0/1629127331 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:54:48.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.493+0000 7ffa73fff700 1 --2- 172.21.15.7:0/1629127331 >> v2:172.21.15.50:3300/0 conn(0x7ffa74096e20 0x7ffa740971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.493+0000 7ffa73fff700 10 monclient(hunting): get_auth_request con 0x7ffa74096e20 auth_method 0 2023-09-30T01:54:48.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.493+0000 7ffa73fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.493+0000 7ffa73fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.493+0000 7ffa73fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.494+0000 7ffa73fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.494+0000 7ffa73fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.494+0000 7ffa73fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.494+0000 7ffa73fff700 10 monclient(hunting): handle_auth_done global_id 4252 payload 274 2023-09-30T01:54:48.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.494+0000 7ffa73fff700 1 -- 172.21.15.7:0/1629127331 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa74096a50 msgr2=0x7ffa74132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.494+0000 7ffa73fff700 1 --2- 172.21.15.7:0/1629127331 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa74096a50 0x7ffa74132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.494+0000 7ffa73fff700 1 -- 172.21.15.7:0/1629127331 >> v1:172.21.15.7:6789/0 conn(0x7ffa7412ef00 legacy=0x7ffa7412f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.494+0000 7ffa73fff700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.494+0000 7ffa73fff700 1 monclient: found mon.noname-c 2023-09-30T01:54:48.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.494+0000 7ffa73fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.494+0000 7ffa73fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3300/0 2023-09-30T01:54:48.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.494+0000 7ffa73fff700 1 -- 172.21.15.7:0/1629127331 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa74099720 con 0x7ffa74096e20 2023-09-30T01:54:48.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.496+0000 7f47ac565700 10 monclient: get_monmap_and_config 2023-09-30T01:54:48.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.496+0000 7f47ac565700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.500 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.496+0000 7f47ac565700 10 monclient: monmap: 2023-09-30T01:54:48.500 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.500 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.500 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:48.497344+0000 2023-09-30T01:54:48.501 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:48.497344+0000 2023-09-30T01:54:48.501 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.501 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.501 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:48.501 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:48.501 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:48.501 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:48.501 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:48.501 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.496+0000 7ffa73fff700 1 -- 172.21.15.7:0/1629127331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa74096e20 msgr2=0x7ffa740971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.496+0000 7ffa73fff700 1 --2- 172.21.15.7:0/1629127331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa74096e20 0x7ffa740971f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ffa600098e0 tx=0x7ffa600091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86e1fd3421e722ee server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.496+0000 7ffa737fe700 1 -- 172.21.15.7:0/1629127331 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6000ba30 con 0x7ffa74096e20 2023-09-30T01:54:48.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.496+0000 7ffa737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.496+0000 7ffa737fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:48.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.496+0000 7ffa737fe700 10 monclient: dump: 2023-09-30T01:54:48.503 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.503 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.503 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.503 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.503 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.503 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.503 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.504 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.504 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.504 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.496+0000 7ffa737fe700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.496+0000 7ffa737fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.496+0000 7ffa737fe700 1 -- 172.21.15.7:0/1629127331 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa6000bb90 con 0x7ffa74096e20 2023-09-30T01:54:48.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.496+0000 7ffa7b08f700 5 monclient: authenticate success, global_id 4252 2023-09-30T01:54:48.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.496+0000 7ffa7b08f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:48.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.496+0000 7ffa737fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.496+0000 7ffa7b08f700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:48.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.496+0000 7ffa7b08f700 4 set_mon_vals no callback set 2023-09-30T01:54:48.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.496+0000 7ffa737fe700 1 -- 172.21.15.7:0/1629127331 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa60011420 con 0x7ffa74096e20 2023-09-30T01:54:48.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.496+0000 7ffa737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.496+0000 7ffa737fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:54:48.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.496+0000 7ffa737fe700 10 monclient: dump: 2023-09-30T01:54:48.505 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.505 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.505 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.505 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.505 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.505 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.505 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.505 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.505 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.505 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7f47ac565700 1 Processor -- start 2023-09-30T01:54:48.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7f47ac565700 1 -- start start 2023-09-30T01:54:48.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7f47ac565700 10 monclient: init 2023-09-30T01:54:48.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7ffa7b08f700 10 monclient: shutdown 2023-09-30T01:54:48.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7ffa7b08f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7ffa7b08f700 1 -- 172.21.15.7:0/1629127331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa74096e20 msgr2=0x7ffa740971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7ffa7b08f700 1 --2- 172.21.15.7:0/1629127331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa74096e20 0x7ffa740971f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ffa600098e0 tx=0x7ffa600091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7f47ac565700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7f47ac565700 10 monclient: _add_conns ranks=[3,4,2,1,0] 2023-09-30T01:54:48.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7f47ac565700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f47a409d250 0x7f47a409caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7f47ac565700 10 monclient(hunting): picked mon.noname-a con 0x7f47a409d250 addr v2:172.21.15.7:3300/0 2023-09-30T01:54:48.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7f47ac565700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f47a409dd00 0x7f47a409e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7f47ac565700 10 monclient(hunting): picked mon.noname-c con 0x7f47a409dd00 addr v2:172.21.15.50:3300/0 2023-09-30T01:54:48.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7f47ac565700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f47a409e660 0x7f47a4136b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7f47ac565700 10 monclient(hunting): picked mon.noname-b con 0x7f47a409e660 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:48.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7f47ac565700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7f47ac565700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a409bde0 con 0x7f47a409d250 2023-09-30T01:54:48.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7f47ac565700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7f47ac565700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a409bb00 con 0x7f47a409dd00 2023-09-30T01:54:48.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7f47ac565700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7f47ac565700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a409aa70 con 0x7f47a409e660 2023-09-30T01:54:48.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7f47ac565700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7f47ac565700 10 monclient(hunting): authenticate will time out at 1177.562901s 2023-09-30T01:54:48.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7ffa7b08f700 1 -- 172.21.15.7:0/1629127331 shutdown_connections 2023-09-30T01:54:48.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7ffa7b08f700 1 --2- 172.21.15.7:0/1629127331 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa74096a50 0x7ffa74132680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7ffa7b08f700 1 --2- 172.21.15.7:0/1629127331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa74096e20 0x7ffa740971f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.497+0000 7ffa7b08f700 1 -- 172.21.15.7:0/1629127331 >> 172.21.15.7:0/1629127331 conn(0x7ffa74070cf0 msgr2=0x7ffa740710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aa301700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f47a409d250 0x7f47a409caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aa301700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f47a409d250 0x7f47a409caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54772/0 (socket says 172.21.15.7:54772) 2023-09-30T01:54:48.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aa301700 1 -- 172.21.15.7:0/3280084408 learned_addr learned my addr 172.21.15.7:0/3280084408 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:48.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aa301700 10 monclient(hunting): get_auth_request con 0x7f47a409d250 auth_method 0 2023-09-30T01:54:48.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aa301700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aa301700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aa301700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7ffa7b08f700 1 -- 172.21.15.7:0/1629127331 shutdown_connections 2023-09-30T01:54:48.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aa301700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aa301700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aa301700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aab02700 1 --2- 172.21.15.7:0/3280084408 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f47a409e660 0x7f47a4136b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7ffa7b08f700 1 -- 172.21.15.7:0/1629127331 wait complete. 2023-09-30T01:54:48.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7ffa7b08f700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7ffa7b08f700 10 monclient: monmap: 2023-09-30T01:54:48.513 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.513 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:48.513 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:48.513 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:48.513 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.513 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.513 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:48.513 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:48.513 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:48.513 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.499+0000 7ffa7b08f700 1 Processor -- start 2023-09-30T01:54:48.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aab02700 10 monclient(hunting): get_auth_request con 0x7f47a409e660 auth_method 0 2023-09-30T01:54:48.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aab02700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aab02700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aab02700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aab02700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aab02700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aab02700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aa301700 10 monclient(hunting): handle_auth_done global_id 4257 payload 274 2023-09-30T01:54:48.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aa301700 1 -- 172.21.15.7:0/3280084408 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f47a409e660 msgr2=0x7f47a4136b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aa301700 1 --2- 172.21.15.7:0/3280084408 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f47a409e660 0x7f47a4136b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aa301700 1 -- 172.21.15.7:0/3280084408 >> v2:172.21.15.50:3300/0 conn(0x7f47a409dd00 msgr2=0x7f47a409e120 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-09-30T01:54:48.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aa301700 1 --2- 172.21.15.7:0/3280084408 >> v2:172.21.15.50:3300/0 conn(0x7f47a409dd00 0x7f47a409e120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aa301700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aa301700 1 monclient: found mon.noname-a 2023-09-30T01:54:48.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aa301700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aa301700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:48.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.498+0000 7f47aa301700 1 -- 172.21.15.7:0/3280084408 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a409c0c0 con 0x7f47a409d250 2023-09-30T01:54:48.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.499+0000 7ffa7b08f700 1 -- start start 2023-09-30T01:54:48.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.499+0000 7ffa7b08f700 10 monclient: init 2023-09-30T01:54:48.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa7b08f700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa7b08f700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:54:48.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa7b08f700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa74096a50 0x7ffa7412df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa7b08f700 10 monclient(hunting): picked mon.noname-c con 0x7ffa74096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:48.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa7b08f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa74096e20 0x7ffa7409da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa7b08f700 10 monclient(hunting): picked mon.noname-b con 0x7ffa74096e20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:48.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa7b08f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa7412ef00 0x7ffa7409dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa7b08f700 10 monclient(hunting): picked mon.noname-a con 0x7ffa7412ef00 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:48.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa7b08f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa7b08f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa740a1890 con 0x7ffa7412ef00 2023-09-30T01:54:48.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa7b08f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa7b08f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7409fc90 con 0x7ffa74096e20 2023-09-30T01:54:48.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa7b08f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa7b08f700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa740a02e0 con 0x7ffa74096a50 2023-09-30T01:54:48.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa7b08f700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa7b08f700 10 monclient(hunting): authenticate will time out at 1177.565160s 2023-09-30T01:54:48.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa78e2b700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa74096a50 0x7ffa7412df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa78e2b700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa74096a50 0x7ffa7412df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59428/0 (socket says 172.21.15.7:59428) 2023-09-30T01:54:48.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa78e2b700 1 -- 172.21.15.7:0/2299502451 learned_addr learned my addr 172.21.15.7:0/2299502451 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:48.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa78e2b700 10 monclient(hunting): get_auth_request con 0x7ffa74096a50 auth_method 0 2023-09-30T01:54:48.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa78e2b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa78e2b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa78e2b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa78e2b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa78e2b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa78e2b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa78e2b700 10 monclient(hunting): handle_auth_done global_id 4256 payload 824 2023-09-30T01:54:48.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa78e2b700 1 -- 172.21.15.7:0/2299502451 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa74096e20 msgr2=0x7ffa7409da70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-09-30T01:54:48.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa78e2b700 1 --2- 172.21.15.7:0/2299502451 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa74096e20 0x7ffa7409da70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa78e2b700 1 -- 172.21.15.7:0/2299502451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa7412ef00 msgr2=0x7ffa7409dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa78e2b700 1 --2- 172.21.15.7:0/2299502451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa7412ef00 0x7ffa7409dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa78e2b700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa78e2b700 1 monclient: found mon.noname-c 2023-09-30T01:54:48.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa78e2b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa78e2b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7ffa78e2b700 1 -- 172.21.15.7:0/2299502451 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa60009420 con 0x7ffa74096a50 2023-09-30T01:54:48.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7f47aa301700 1 -- 172.21.15.7:0/3280084408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47a409d250 msgr2=0x7f47a409caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.500+0000 7f47aa301700 1 --2- 172.21.15.7:0/3280084408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47a409d250 0x7f47a409caa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f47a40a0e30 tx=0x7f47a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b30ebc23cc8b9344 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47a92ff700 1 -- 172.21.15.7:0/3280084408 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a0009e70 con 0x7f47a409d250 2023-09-30T01:54:48.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47a92ff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47a92ff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:48.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47a92ff700 10 monclient: dump: 2023-09-30T01:54:48.523 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.523 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.523 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.524 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.524 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.524 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.524 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.524 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.524 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.524 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47a92ff700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47a92ff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47ac565700 5 monclient: authenticate success, global_id 4257 2023-09-30T01:54:48.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47ac565700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:48.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47a92ff700 1 -- 172.21.15.7:0/3280084408 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a000ba70 con 0x7f47a409d250 2023-09-30T01:54:48.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47a92ff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47ac565700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:48.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47a92ff700 1 -- 172.21.15.7:0/3280084408 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a000be30 con 0x7f47a409d250 2023-09-30T01:54:48.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47a92ff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47ac565700 4 set_mon_vals no callback set 2023-09-30T01:54:48.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47a92ff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:48.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47a92ff700 10 monclient: dump: 2023-09-30T01:54:48.525 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.525 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.525 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.525 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.526 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.526 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.526 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.526 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.526 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.526 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47ac565700 10 monclient: shutdown 2023-09-30T01:54:48.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47ac565700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47ac565700 1 -- 172.21.15.7:0/3280084408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47a409d250 msgr2=0x7f47a409caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47ac565700 1 --2- 172.21.15.7:0/3280084408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47a409d250 0x7f47a409caa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f47a40a0e30 tx=0x7f47a0009250 comp rx=0 tx=0).stop 2023-09-30T01:54:48.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47ac565700 1 -- 172.21.15.7:0/3280084408 shutdown_connections 2023-09-30T01:54:48.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47ac565700 1 --2- 172.21.15.7:0/3280084408 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f47a409e660 0x7f47a4136b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47ac565700 1 --2- 172.21.15.7:0/3280084408 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47a409d250 0x7f47a409caa0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.501+0000 7f47ac565700 1 --2- 172.21.15.7:0/3280084408 >> v2:172.21.15.50:3300/0 conn(0x7f47a409dd00 0x7f47a409e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.502+0000 7f47ac565700 1 -- 172.21.15.7:0/3280084408 >> 172.21.15.7:0/3280084408 conn(0x7f47a4071910 msgr2=0x7f47a409fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.502+0000 7f47ac565700 1 -- 172.21.15.7:0/3280084408 shutdown_connections 2023-09-30T01:54:48.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.502+0000 7f47ac565700 1 -- 172.21.15.7:0/3280084408 wait complete. 2023-09-30T01:54:48.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.502+0000 7f47ac565700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.502+0000 7f47ac565700 10 monclient: monmap: 2023-09-30T01:54:48.527 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.527 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:48.527 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:48.527 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:48.527 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.528 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.528 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:48.528 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:48.528 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:48.528 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.502+0000 7ffa78e2b700 1 -- 172.21.15.7:0/2299502451 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa74096a50 msgr2=0x7ffa7412df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.502+0000 7ffa78e2b700 1 --2- 172.21.15.7:0/2299502451 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa74096a50 0x7ffa7412df80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ffa6800b870 tx=0x7ffa68009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.502+0000 7f47ac565700 1 Processor -- start 2023-09-30T01:54:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7ffa78e2b700 1 --2- 172.21.15.7:0/2299502451 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa74096a50 0x7ffa7412df80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ffa6800b870 tx=0x7ffa68009250 comp rx=0 tx=0).queue_received will delay after 1.69604e+09 on 0x7ffa6800fdc0 mon_map magic: 0 v1 2023-09-30T01:54:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7f47ac565700 1 -- start start 2023-09-30T01:54:48.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7f47ac565700 10 monclient: init 2023-09-30T01:54:48.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7ffa70ff9700 1 -- 172.21.15.7:0/2299502451 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6800f880 con 0x7ffa74096a50 2023-09-30T01:54:48.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7ffa70ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7ffa70ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:48.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7ffa70ff9700 10 monclient: dump: 2023-09-30T01:54:48.530 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.530 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.530 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.531 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7ffa70ff9700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7ffa70ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7ffa70ff9700 1 -- 172.21.15.7:0/2299502451 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa6800fa00 con 0x7ffa74096a50 2023-09-30T01:54:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7ffa70ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7ffa7b08f700 5 monclient: authenticate success, global_id 4256 2023-09-30T01:54:48.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7ffa7b08f700 10 monclient: _renew_subs 2023-09-30T01:54:48.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7ffa7b08f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7ffa7b08f700 1 -- 172.21.15.7:0/2299502451 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa741334e0 con 0x7ffa74096a50 2023-09-30T01:54:48.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7ffa7b08f700 10 monclient: _renew_subs 2023-09-30T01:54:48.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7ffa7b08f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7ffa7b08f700 1 -- 172.21.15.7:0/2299502451 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa7409f6a0 con 0x7ffa74096a50 2023-09-30T01:54:48.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7ffa72ffd700 4 set_mon_vals no callback set 2023-09-30T01:54:48.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7f47ac565700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7f47ac565700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:54:48.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7f47ac565700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47a409d250 0x7f47a413c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7f47ac565700 10 monclient(hunting): picked mon.noname-a con 0x7f47a409d250 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:48.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7f47ac565700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47a409dd00 0x7f47a413b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7f47ac565700 10 monclient(hunting): picked mon.noname-b con 0x7f47a409dd00 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:48.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7f47ac565700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f47a409e660 0x7f47a413f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7f47ac565700 10 monclient(hunting): picked mon.noname-c con 0x7f47a409e660 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:48.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7f47ac565700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7f47ac565700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a4120bb0 con 0x7f47a409d250 2023-09-30T01:54:48.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7f47ac565700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7f47ac565700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a411eea0 con 0x7f47a409dd00 2023-09-30T01:54:48.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7f47ac565700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7f47ac565700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a41208d0 con 0x7f47a409e660 2023-09-30T01:54:48.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7f47ac565700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.503+0000 7f47ac565700 10 monclient(hunting): authenticate will time out at 1177.568856s 2023-09-30T01:54:48.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47aab02700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f47a409e660 0x7f47a413f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47a9b00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47a409dd00 0x7f47a413b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47aab02700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f47a409e660 0x7f47a413f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59432/0 (socket says 172.21.15.7:59432) 2023-09-30T01:54:48.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47aab02700 1 -- 172.21.15.7:0/762937196 learned_addr learned my addr 172.21.15.7:0/762937196 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:48.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47aab02700 10 monclient(hunting): get_auth_request con 0x7f47a409e660 auth_method 0 2023-09-30T01:54:48.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47a9b00700 1 --2- 172.21.15.7:0/762937196 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47a409dd00 0x7f47a413b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.7:47894/0 (socket says 172.21.15.7:47894) 2023-09-30T01:54:48.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47aab02700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47aab02700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47aab02700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47a9b00700 10 monclient(hunting): get_auth_request con 0x7f47a409dd00 auth_method 0 2023-09-30T01:54:48.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47a9b00700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47a9b00700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47a9b00700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47a9b00700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47a9b00700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47a9b00700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47aab02700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47aab02700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47aab02700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47aa301700 1 --2- 172.21.15.7:0/762937196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47a409d250 0x7f47a413c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47aa301700 10 monclient(hunting): get_auth_request con 0x7f47a409d250 auth_method 0 2023-09-30T01:54:48.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47aa301700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47aa301700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47aa301700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47aa301700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47aa301700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47aa301700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47a9b00700 10 monclient(hunting): handle_auth_done global_id 4255 payload 824 2023-09-30T01:54:48.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47a9b00700 1 -- 172.21.15.7:0/762937196 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f47a409e660 msgr2=0x7f47a413f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47a9b00700 1 --2- 172.21.15.7:0/762937196 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f47a409e660 0x7f47a413f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47a9b00700 1 -- 172.21.15.7:0/762937196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47a409d250 msgr2=0x7f47a413c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47a9b00700 1 --2- 172.21.15.7:0/762937196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47a409d250 0x7f47a413c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47a9b00700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47a9b00700 1 monclient: found mon.noname-b 2023-09-30T01:54:48.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47a9b00700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47a9b00700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.50:3300/0 2023-09-30T01:54:48.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47a9b00700 1 -- 172.21.15.7:0/762937196 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a00094f0 con 0x7f47a409dd00 2023-09-30T01:54:48.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7f47aab02700 1 --2- 172.21.15.7:0/762937196 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f47a409e660 0x7f47a413f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:48.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7ffa7b08f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:48.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7ffa7b08f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:48.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7ffa7b08f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.504+0000 7ffa7b08f700 1 -- 172.21.15.7:0/2299502451 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa74064c10 con 0x7ffa74096a50 2023-09-30T01:54:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.505+0000 7ffa70ff9700 1 -- 172.21.15.7:0/2299502451 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6800fdc0 con 0x7ffa74096a50 2023-09-30T01:54:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.505+0000 7ffa70ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.505+0000 7ffa70ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.505+0000 7ffa70ff9700 10 monclient: dump: 2023-09-30T01:54:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.540 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.541 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.541 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.541 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.505+0000 7ffa70ff9700 1 -- 172.21.15.7:0/2299502451 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7ffa68010630 con 0x7ffa74096a50 2023-09-30T01:54:48.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.505+0000 7ffa70ff9700 1 --2- 172.21.15.7:0/2299502451 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ffa58035830 0x7ffa58037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.505+0000 7ffa73fff700 1 --2- 172.21.15.7:0/2299502451 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ffa58035830 0x7ffa58037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.506+0000 7ffa73fff700 10 monclient: get_auth_request con 0x7ffa58035830 auth_method 0 2023-09-30T01:54:48.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f47a9b00700 1 -- 172.21.15.7:0/762937196 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47a409dd00 msgr2=0x7f47a413b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f47a9b00700 1 --2- 172.21.15.7:0/762937196 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47a409dd00 0x7f47a413b830 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f479400b5b0 tx=0x7f47940090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f479a7fc700 1 -- 172.21.15.7:0/762937196 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479400f720 con 0x7f47a409dd00 2023-09-30T01:54:48.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f479a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f479a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:54:48.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f479a7fc700 10 monclient: dump: 2023-09-30T01:54:48.543 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.543 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.543 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.543 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.543 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.544 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f479a7fc700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f479a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f479a7fc700 1 -- 172.21.15.7:0/762937196 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f479400f8a0 con 0x7f47a409dd00 2023-09-30T01:54:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f479a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f479a7fc700 1 -- 172.21.15.7:0/762937196 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479400fc60 con 0x7f47a409dd00 2023-09-30T01:54:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f479a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f479a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:54:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f479a7fc700 10 monclient: dump: 2023-09-30T01:54:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.545 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f47a8afe700 4 set_mon_vals no callback set 2023-09-30T01:54:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f47ac565700 5 monclient: authenticate success, global_id 4255 2023-09-30T01:54:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f47ac565700 10 monclient: _renew_subs 2023-09-30T01:54:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f47ac565700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f47ac565700 1 -- 172.21.15.7:0/762937196 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a4120e90 con 0x7f47a409dd00 2023-09-30T01:54:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f47ac565700 10 monclient: _renew_subs 2023-09-30T01:54:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f47ac565700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.507+0000 7f47ac565700 1 -- 172.21.15.7:0/762937196 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a413bf30 con 0x7f47a409dd00 2023-09-30T01:54:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.508+0000 7f479a7fc700 1 -- 172.21.15.7:0/762937196 <== mon.1 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f4794011610 con 0x7f47a409dd00 2023-09-30T01:54:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.508+0000 7f479a7fc700 1 --2- 172.21.15.7:0/762937196 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f4788035830 0x7f4788037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.508+0000 7f479a7fc700 1 -- 172.21.15.7:0/762937196 <== mon.1 v2:172.21.15.50:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f47940474e0 con 0x7f47a409dd00 2023-09-30T01:54:48.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.509+0000 7ffa73fff700 1 --2- 172.21.15.7:0/2299502451 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ffa58035830 0x7ffa58037ce0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ffa7409f2a0 tx=0x7ffa60016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.509+0000 7f47aa301700 1 --2- 172.21.15.7:0/762937196 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f4788035830 0x7f4788037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.509+0000 7f47aa301700 10 monclient: get_auth_request con 0x7f4788035830 auth_method 0 2023-09-30T01:54:48.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.510+0000 7f47ac565700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:48.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.510+0000 7f47ac565700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:48.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.510+0000 7f47ac565700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:48.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.510+0000 7f47ac565700 1 -- 172.21.15.7:0/762937196 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a4064c10 con 0x7f47a409dd00 2023-09-30T01:54:48.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.510+0000 7ffa70ff9700 1 -- 172.21.15.7:0/2299502451 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7ffa680487a0 con 0x7ffa74096a50 2023-09-30T01:54:48.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.511+0000 7f47aa301700 1 --2- 172.21.15.7:0/762937196 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f4788035830 0x7f4788037ce0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f47a40a0e30 tx=0x7f47a0017000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.515+0000 7f479a7fc700 1 -- 172.21.15.7:0/762937196 <== mon.1 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f479400fdc0 con 0x7f47a409dd00 2023-09-30T01:54:48.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.516+0000 7f479a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:48.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.516+0000 7f479a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:48.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.540+0000 7eff9da47700 10 monclient: get_monmap_and_config 2023-09-30T01:54:48.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.540+0000 7eff9da47700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.540+0000 7eff9da47700 10 monclient: monmap: 2023-09-30T01:54:48.552 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.552 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.552 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:48.541693+0000 2023-09-30T01:54:48.552 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:48.541693+0000 2023-09-30T01:54:48.552 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.553 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.553 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:48.553 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:48.553 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:48.553 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:48.553 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:48.553 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.541+0000 7eff9da47700 1 Processor -- start 2023-09-30T01:54:48.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.541+0000 7eff9da47700 1 -- start start 2023-09-30T01:54:48.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.541+0000 7eff9da47700 10 monclient: init 2023-09-30T01:54:48.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff9da47700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff9da47700 10 monclient: _add_conns ranks=[0,4,2,3,1] 2023-09-30T01:54:48.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff9da47700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff98070cf0 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:48.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff9da47700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7eff9806e470 0x7eff981344b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff9da47700 10 monclient(hunting): picked mon.noname-c con 0x7eff9806e470 addr v2:172.21.15.50:3300/0 2023-09-30T01:54:48.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff9da47700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7eff9806e840 0x7eff98136a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff9da47700 10 monclient(hunting): picked mon.noname-b con 0x7eff9806e840 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:48.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff9da47700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff980a14f0 con 0x7eff98070cf0 2023-09-30T01:54:48.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff9da47700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff9da47700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff981300a0 con 0x7eff9806e470 2023-09-30T01:54:48.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff9da47700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff9da47700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff9806c980 con 0x7eff9806e840 2023-09-30T01:54:48.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff9da47700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff9da47700 10 monclient(hunting): authenticate will time out at 1177.607369s 2023-09-30T01:54:48.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff96ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7eff98070cf0 0x7eff9812f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58102/0 (socket says 172.21.15.7:58102) 2023-09-30T01:54:48.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff96ffd700 1 -- 172.21.15.7:0/125914590 learned_addr learned my addr 172.21.15.7:0/125914590 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:54:48.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff977fe700 1 --2- 172.21.15.7:0/125914590 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7eff9806e840 0x7eff98136a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff977fe700 10 monclient(hunting): get_auth_request con 0x7eff9806e840 auth_method 0 2023-09-30T01:54:48.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff977fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff977fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff977fe700 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2023-09-30T01:54:48.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff977fe700 1 -- 172.21.15.7:0/125914590 >> v2:172.21.15.50:3300/0 conn(0x7eff9806e470 msgr2=0x7eff981344b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff977fe700 1 --2- 172.21.15.7:0/125914590 >> v2:172.21.15.50:3300/0 conn(0x7eff9806e470 0x7eff981344b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff977fe700 1 -- 172.21.15.7:0/125914590 >> v1:172.21.15.7:6789/0 conn(0x7eff98070cf0 legacy=0x7eff9812f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.542+0000 7eff977fe700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.543+0000 7eff977fe700 1 monclient: found mon.noname-b 2023-09-30T01:54:48.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.543+0000 7eff977fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.543+0000 7eff977fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.543+0000 7eff977fe700 1 -- 172.21.15.7:0/125914590 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff980710c0 con 0x7eff9806e840 2023-09-30T01:54:48.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.545+0000 7eff977fe700 1 -- 172.21.15.7:0/125914590 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7eff9806e840 msgr2=0x7eff98136a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.545+0000 7eff977fe700 1 --2- 172.21.15.7:0/125914590 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7eff9806e840 0x7eff98136a00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7eff90009ee0 tx=0x7eff90009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36ca05a437498a8d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.545+0000 7eff95ffb700 1 -- 172.21.15.7:0/125914590 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff90011670 con 0x7eff9806e840 2023-09-30T01:54:48.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.545+0000 7eff95ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.545+0000 7eff95ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:54:48.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.545+0000 7eff95ffb700 10 monclient: dump: 2023-09-30T01:54:48.557 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.557 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.557 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.557 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.557 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.557 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.557 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.557 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.558 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.558 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.545+0000 7eff95ffb700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.545+0000 7eff95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.545+0000 7eff9da47700 5 monclient: authenticate success, global_id 4262 2023-09-30T01:54:48.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.545+0000 7eff95ffb700 1 -- 172.21.15.7:0/125914590 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff900117d0 con 0x7eff9806e840 2023-09-30T01:54:48.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.545+0000 7eff95ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.545+0000 7eff9da47700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:48.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.545+0000 7eff95ffb700 1 -- 172.21.15.7:0/125914590 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff90011b90 con 0x7eff9806e840 2023-09-30T01:54:48.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.545+0000 7eff9da47700 4 set_mon_vals no callback set 2023-09-30T01:54:48.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.545+0000 7eff95ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.545+0000 7eff95ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:48.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.545+0000 7eff95ffb700 10 monclient: dump: 2023-09-30T01:54:48.559 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.559 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.559 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.559 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.559 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.559 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.559 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.559 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.559 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.559 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.546+0000 7eff9da47700 10 monclient: shutdown 2023-09-30T01:54:48.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.546+0000 7eff9da47700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.546+0000 7eff9da47700 1 -- 172.21.15.7:0/125914590 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7eff9806e840 msgr2=0x7eff98136a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.546+0000 7eff9da47700 1 --2- 172.21.15.7:0/125914590 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7eff9806e840 0x7eff98136a00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7eff90009ee0 tx=0x7eff90009780 comp rx=0 tx=0).stop 2023-09-30T01:54:48.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.546+0000 7f2e4faab700 10 monclient: get_monmap_and_config 2023-09-30T01:54:48.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.546+0000 7f2e4faab700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.546+0000 7f2e4faab700 10 monclient: monmap: 2023-09-30T01:54:48.561 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.561 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.561 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:48.547413+0000 2023-09-30T01:54:48.561 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:48.547413+0000 2023-09-30T01:54:48.561 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.561 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.561 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:48.561 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:48.561 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:48.561 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:48.561 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:48.561 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.546+0000 7eff9da47700 1 -- 172.21.15.7:0/125914590 shutdown_connections 2023-09-30T01:54:48.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.546+0000 7eff9da47700 1 --2- 172.21.15.7:0/125914590 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7eff9806e840 0x7eff98136a00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.546+0000 7eff9da47700 1 --2- 172.21.15.7:0/125914590 >> v2:172.21.15.50:3300/0 conn(0x7eff9806e470 0x7eff981344b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.546+0000 7eff9da47700 1 -- 172.21.15.7:0/125914590 >> 172.21.15.7:0/125914590 conn(0x7eff98067f20 msgr2=0x7eff980a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.546+0000 7eff9da47700 1 -- 172.21.15.7:0/125914590 shutdown_connections 2023-09-30T01:54:48.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.546+0000 7eff9da47700 1 -- 172.21.15.7:0/125914590 wait complete. 2023-09-30T01:54:48.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.547+0000 7f2e4faab700 1 Processor -- start 2023-09-30T01:54:48.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.547+0000 7f2e4faab700 1 -- start start 2023-09-30T01:54:48.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.547+0000 7f2e4faab700 10 monclient: init 2023-09-30T01:54:48.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.547+0000 7eff9da47700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.547+0000 7eff9da47700 10 monclient: monmap: 2023-09-30T01:54:48.564 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.564 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:48.564 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:48.564 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:48.564 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.564 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.564 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:48.564 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:48.564 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:48.565 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.547+0000 7f2e4faab700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.547+0000 7f2e4faab700 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-09-30T01:54:48.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.547+0000 7f2e4faab700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e48096a50 addr v1:172.21.15.50:6789/0 2023-09-30T01:54:48.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.547+0000 7f2e4faab700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f2e48096e20 0x7f2e48097220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.547+0000 7f2e4faab700 10 monclient(hunting): picked mon.noname-c con 0x7f2e48096e20 addr v2:172.21.15.50:3300/0 2023-09-30T01:54:48.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.547+0000 7f2e4faab700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e48137710 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:48.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.547+0000 7f2e4faab700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e48132d90 con 0x7f2e48137710 2023-09-30T01:54:48.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.547+0000 7f2e4faab700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e48097b40 con 0x7f2e48096a50 2023-09-30T01:54:48.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.547+0000 7f2e4faab700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.547+0000 7f2e4faab700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e48099160 con 0x7f2e48096e20 2023-09-30T01:54:48.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.547+0000 7f2e4faab700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.547+0000 7f2e4faab700 10 monclient(hunting): authenticate will time out at 1177.612852s 2023-09-30T01:54:48.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.547+0000 7f2e4e048700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2e48137710 0x7f2e48139af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58114/0 (socket says 172.21.15.7:58114) 2023-09-30T01:54:48.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.547+0000 7f2e4e048700 1 -- 172.21.15.7:0/498748032 learned_addr learned my addr 172.21.15.7:0/498748032 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:54:48.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7f2e4d046700 1 --2- 172.21.15.7:0/498748032 >> v2:172.21.15.50:3300/0 conn(0x7f2e48096e20 0x7f2e48097220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7eff9da47700 1 Processor -- start 2023-09-30T01:54:48.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7f2e4d046700 10 monclient(hunting): get_auth_request con 0x7f2e48096e20 auth_method 0 2023-09-30T01:54:48.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7f2e4d046700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7f2e4d046700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7f2e4d046700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7eff9da47700 1 -- start start 2023-09-30T01:54:48.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7eff9da47700 10 monclient: init 2023-09-30T01:54:48.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7f2e4d046700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7f2e4d046700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7f2e4d046700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7f2e4d046700 10 monclient(hunting): handle_auth_done global_id 4258 payload 274 2023-09-30T01:54:48.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7f2e4d046700 1 -- 172.21.15.7:0/498748032 >> v1:172.21.15.50:6789/0 conn(0x7f2e48096a50 legacy=0x7f2e48132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7f2e4d046700 1 -- 172.21.15.7:0/498748032 >> v1:172.21.15.7:6789/0 conn(0x7f2e48137710 legacy=0x7f2e48139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7f2e4d046700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7f2e4d046700 1 monclient: found mon.noname-c 2023-09-30T01:54:48.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7f2e4d046700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7f2e4d046700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3300/0 2023-09-30T01:54:48.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7f2e4d046700 1 -- 172.21.15.7:0/498748032 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e48099720 con 0x7f2e48096e20 2023-09-30T01:54:48.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7eff9da47700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7eff9da47700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:54:48.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7eff9da47700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7eff9806e470 0x7eff9813d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7eff9da47700 10 monclient(hunting): picked mon.noname-c con 0x7eff9806e470 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:48.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7eff9da47700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff9806e840 0x7eff9812d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7eff9da47700 10 monclient(hunting): picked mon.noname-a con 0x7eff9806e840 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:48.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7eff9da47700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff98070cf0 0x7eff9812d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7eff9da47700 10 monclient(hunting): picked mon.noname-b con 0x7eff98070cf0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:48.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7eff9da47700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7eff9da47700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9812f220 con 0x7eff9806e840 2023-09-30T01:54:48.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7eff9da47700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7eff9da47700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9811b6e0 con 0x7eff98070cf0 2023-09-30T01:54:48.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7eff9da47700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7eff9da47700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff9813df30 con 0x7eff9806e470 2023-09-30T01:54:48.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7eff9da47700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.548+0000 7eff9da47700 10 monclient(hunting): authenticate will time out at 1177.613892s 2023-09-30T01:54:48.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.549+0000 7eff96ffd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7eff9806e470 0x7eff9813d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.549+0000 7eff96ffd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7eff9806e470 0x7eff9813d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59442/0 (socket says 172.21.15.7:59442) 2023-09-30T01:54:48.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.549+0000 7eff96ffd700 1 -- 172.21.15.7:0/2269392554 learned_addr learned my addr 172.21.15.7:0/2269392554 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:48.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.549+0000 7eff96ffd700 10 monclient(hunting): get_auth_request con 0x7eff9806e470 auth_method 0 2023-09-30T01:54:48.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.549+0000 7eff96ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.549+0000 7eff96ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.549+0000 7eff96ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.549+0000 7eff96ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.549+0000 7eff96ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.549+0000 7eff96ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.549+0000 7eff96ffd700 10 monclient(hunting): handle_auth_done global_id 4265 payload 824 2023-09-30T01:54:48.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.549+0000 7eff96ffd700 1 -- 172.21.15.7:0/2269392554 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff98070cf0 msgr2=0x7eff9812d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.549+0000 7eff96ffd700 1 --2- 172.21.15.7:0/2269392554 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff98070cf0 0x7eff9812d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.549+0000 7eff96ffd700 1 -- 172.21.15.7:0/2269392554 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff9806e840 msgr2=0x7eff9812d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.549+0000 7eff96ffd700 1 --2- 172.21.15.7:0/2269392554 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff9806e840 0x7eff9812d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.549+0000 7eff96ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.549+0000 7eff96ffd700 1 monclient: found mon.noname-c 2023-09-30T01:54:48.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.549+0000 7eff96ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.549+0000 7eff96ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.549+0000 7eff96ffd700 1 -- 172.21.15.7:0/2269392554 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff90009a20 con 0x7eff9806e470 2023-09-30T01:54:48.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.550+0000 7f2e4d046700 1 -- 172.21.15.7:0/498748032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e48096e20 msgr2=0x7f2e48097220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.550+0000 7f2e4d046700 1 --2- 172.21.15.7:0/498748032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e48096e20 0x7f2e48097220 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2e380098e0 tx=0x7f2e380091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6ca2e12ac0887b7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f2e4c845700 1 -- 172.21.15.7:0/498748032 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3800b9f0 con 0x7f2e48096e20 2023-09-30T01:54:48.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f2e4c845700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f2e4c845700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:48.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f2e4c845700 10 monclient: dump: 2023-09-30T01:54:48.576 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.576 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.576 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.576 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.576 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.576 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.576 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.576 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.577 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.577 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f2e4c845700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f2e4c845700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f2e4faab700 5 monclient: authenticate success, global_id 4258 2023-09-30T01:54:48.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f2e4c845700 1 -- 172.21.15.7:0/498748032 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3800bb50 con 0x7f2e48096e20 2023-09-30T01:54:48.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f2e4faab700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:48.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f2e4c845700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f2e4faab700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:48.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f2e4faab700 4 set_mon_vals no callback set 2023-09-30T01:54:48.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f2e4c845700 1 -- 172.21.15.7:0/498748032 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e38011420 con 0x7f2e48096e20 2023-09-30T01:54:48.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f2e4c845700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f2e4c845700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:54:48.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f2e4c845700 10 monclient: dump: 2023-09-30T01:54:48.578 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.578 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.578 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.578 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.578 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.578 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.578 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.578 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.579 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.579 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f15d8de6700 10 monclient: get_monmap_and_config 2023-09-30T01:54:48.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f15d8de6700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f15d8de6700 10 monclient: monmap: 2023-09-30T01:54:48.579 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.579 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.579 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:48.552431+0000 2023-09-30T01:54:48.580 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:48.552431+0000 2023-09-30T01:54:48.580 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.580 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.580 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:48.580 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:48.580 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:48.580 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:48.580 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:48.580 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7eff96ffd700 1 -- 172.21.15.7:0/2269392554 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7eff9806e470 msgr2=0x7eff9813d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7eff96ffd700 1 --2- 172.21.15.7:0/2269392554 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7eff9806e470 0x7eff9813d530 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7eff8800c5e0 tx=0x7eff88004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f919a8f4700 10 monclient: get_monmap_and_config 2023-09-30T01:54:48.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f919a8f4700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7f919a8f4700 10 monclient: monmap: 2023-09-30T01:54:48.581 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.581 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.582 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:48.552917+0000 2023-09-30T01:54:48.582 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:48.552917+0000 2023-09-30T01:54:48.582 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.582 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.582 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:48.582 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:48.582 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:48.582 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:48.582 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:48.582 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7eff7effd700 1 -- 172.21.15.7:0/2269392554 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff88005d40 con 0x7eff9806e470 2023-09-30T01:54:48.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7eff7effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7eff7effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:48.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7eff7effd700 10 monclient: dump: 2023-09-30T01:54:48.583 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.583 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.583 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.583 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.583 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.584 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.584 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.584 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.584 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.584 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7eff7effd700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7eff7effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7eff9da47700 5 monclient: authenticate success, global_id 4265 2023-09-30T01:54:48.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7eff7effd700 1 -- 172.21.15.7:0/2269392554 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff880070d0 con 0x7eff9806e470 2023-09-30T01:54:48.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7eff7effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7eff7effd700 1 -- 172.21.15.7:0/2269392554 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff88007490 con 0x7eff9806e470 2023-09-30T01:54:48.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7eff7effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7eff7effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:48.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7eff7effd700 10 monclient: dump: 2023-09-30T01:54:48.585 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.585 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.585 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.585 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.585 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.585 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.585 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.585 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.585 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.585 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.551+0000 7eff957fa700 4 set_mon_vals no callback set 2023-09-30T01:54:48.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f15d8de6700 1 Processor -- start 2023-09-30T01:54:48.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7eff9da47700 10 monclient: _renew_subs 2023-09-30T01:54:48.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7eff9da47700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7eff9da47700 1 -- 172.21.15.7:0/2269392554 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff9811b9c0 con 0x7eff9806e470 2023-09-30T01:54:48.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7eff9da47700 10 monclient: _renew_subs 2023-09-30T01:54:48.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7eff9da47700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7eff9da47700 1 -- 172.21.15.7:0/2269392554 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff9813da70 con 0x7eff9806e470 2023-09-30T01:54:48.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f15d8de6700 1 -- start start 2023-09-30T01:54:48.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f15d8de6700 10 monclient: init 2023-09-30T01:54:48.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f919a8f4700 1 Processor -- start 2023-09-30T01:54:48.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f919a8f4700 1 -- start start 2023-09-30T01:54:48.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f919a8f4700 10 monclient: init 2023-09-30T01:54:48.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f15d8de6700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f15d8de6700 10 monclient: _add_conns ranks=[3,2,0,4,1] 2023-09-30T01:54:48.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f2e4faab700 10 monclient: shutdown 2023-09-30T01:54:48.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f15d8de6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f15d406ebd0 0x7f15d409b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f2e4faab700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f2e4faab700 1 -- 172.21.15.7:0/498748032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e48096e20 msgr2=0x7f2e48097220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f2e4faab700 1 --2- 172.21.15.7:0/498748032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e48096e20 0x7f2e48097220 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2e380098e0 tx=0x7f2e380091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f2e4faab700 1 -- 172.21.15.7:0/498748032 shutdown_connections 2023-09-30T01:54:48.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f2e4faab700 1 --2- 172.21.15.7:0/498748032 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e48096e20 0x7f2e48097220 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f15d8de6700 10 monclient(hunting): picked mon.noname-a con 0x7f15d406ebd0 addr v2:172.21.15.7:3300/0 2023-09-30T01:54:48.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f15d8de6700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f15d409c7a0 0x7f15d409cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f2e4faab700 1 -- 172.21.15.7:0/498748032 >> 172.21.15.7:0/498748032 conn(0x7f2e48070cf0 msgr2=0x7f2e480710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f15d8de6700 10 monclient(hunting): picked mon.noname-b con 0x7f15d409c7a0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:48.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f15d8de6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f15d409d0e0 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:48.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f15d8de6700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d409e570 con 0x7f15d409d0e0 2023-09-30T01:54:48.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f15d8de6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f15d8de6700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15d409dc10 con 0x7f15d406ebd0 2023-09-30T01:54:48.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f15d8de6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.552+0000 7f15d8de6700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d4097f40 con 0x7f15d409c7a0 2023-09-30T01:54:48.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f15d8de6700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f15d8de6700 10 monclient(hunting): authenticate will time out at 1177.618015s 2023-09-30T01:54:48.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f15d259c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f15d406ebd0 0x7f15d409b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f15d2d9d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f15d409d0e0 0x7f15d41355b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58118/0 (socket says 172.21.15.7:58118) 2023-09-30T01:54:48.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f15d2d9d700 1 -- 172.21.15.7:0/3737738535 learned_addr learned my addr 172.21.15.7:0/3737738535 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:54:48.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f15d259c700 10 monclient(hunting): get_auth_request con 0x7f15d406ebd0 auth_method 0 2023-09-30T01:54:48.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f15d259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f15d259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f15d259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f2e4faab700 1 -- 172.21.15.7:0/498748032 shutdown_connections 2023-09-30T01:54:48.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f15d259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f15d259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f15d259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f2e4faab700 1 -- 172.21.15.7:0/498748032 wait complete. 2023-09-30T01:54:48.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7eff9da47700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:48.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7eff9da47700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:48.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7eff9da47700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f15d259c700 10 monclient(hunting): handle_auth_done global_id 4263 payload 274 2023-09-30T01:54:48.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f15d259c700 1 -- 172.21.15.7:0/3737738535 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f15d409c7a0 msgr2=0x7f15d409cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f15d259c700 1 --2- 172.21.15.7:0/3737738535 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f15d409c7a0 0x7f15d409cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f15d259c700 1 -- 172.21.15.7:0/3737738535 >> v1:172.21.15.7:6789/0 conn(0x7f15d409d0e0 legacy=0x7f15d41355b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f15d259c700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f15d259c700 1 monclient: found mon.noname-a 2023-09-30T01:54:48.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f15d259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f15d259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:48.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f15d259c700 1 -- 172.21.15.7:0/3737738535 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d4096f10 con 0x7f15d406ebd0 2023-09-30T01:54:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f919a8f4700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f919a8f4700 10 monclient: _add_conns ranks=[0,3,4,1,2] 2023-09-30T01:54:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f919a8f4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f919409de10 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f919a8f4700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f919409e1e0 0x7f919409e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f919a8f4700 10 monclient(hunting): picked mon.noname-a con 0x7f919409e1e0 addr v2:172.21.15.7:3300/0 2023-09-30T01:54:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f919a8f4700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9194137840 0x7f9194139c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f919a8f4700 10 monclient(hunting): picked mon.noname-c con 0x7f9194137840 addr v2:172.21.15.50:3300/0 2023-09-30T01:54:48.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f919a8f4700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919406a0a0 con 0x7f919409de10 2023-09-30T01:54:48.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f2e4faab700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f919a8f4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f919a8f4700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f919406b440 con 0x7f919409e1e0 2023-09-30T01:54:48.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f919a8f4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f919a8f4700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9194069dc0 con 0x7f9194137840 2023-09-30T01:54:48.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f919a8f4700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f919a8f4700 10 monclient(hunting): authenticate will time out at 1177.618559s 2023-09-30T01:54:48.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f91937fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f919409e1e0 0x7f919409e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f91937fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f919409e1e0 0x7f919409e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54816/0 (socket says 172.21.15.7:54816) 2023-09-30T01:54:48.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f91937fe700 1 -- 172.21.15.7:0/3536293837 learned_addr learned my addr 172.21.15.7:0/3536293837 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:48.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f91937fe700 10 monclient(hunting): get_auth_request con 0x7f919409e1e0 auth_method 0 2023-09-30T01:54:48.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f2e4faab700 10 monclient: monmap: 2023-09-30T01:54:48.597 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.597 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:48.597 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:48.597 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:48.597 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.598 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.598 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:48.598 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:48.598 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:48.598 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f91937fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f91937fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f91937fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f91937fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f91937fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7f91937fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f9198e91700 1 --2- 172.21.15.7:0/3536293837 >> v2:172.21.15.50:3300/0 conn(0x7f9194137840 0x7f9194139c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f9198e91700 10 monclient(hunting): get_auth_request con 0x7f9194137840 auth_method 0 2023-09-30T01:54:48.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f9198e91700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f9198e91700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f9198e91700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.553+0000 7eff9da47700 1 -- 172.21.15.7:0/2269392554 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff98064c10 con 0x7eff9806e470 2023-09-30T01:54:48.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7eff7effd700 1 -- 172.21.15.7:0/2269392554 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7eff8801b3b0 con 0x7eff9806e470 2023-09-30T01:54:48.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f9198e91700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f9198e91700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f9198e91700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f9198e91700 10 monclient(hunting): handle_auth_done global_id 4261 payload 274 2023-09-30T01:54:48.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f9198e91700 1 -- 172.21.15.7:0/3536293837 >> v2:172.21.15.7:3300/0 conn(0x7f919409e1e0 msgr2=0x7f919409e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f9198e91700 1 --2- 172.21.15.7:0/3536293837 >> v2:172.21.15.7:3300/0 conn(0x7f919409e1e0 0x7f919409e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f9198e91700 1 -- 172.21.15.7:0/3536293837 >> v1:172.21.15.7:6789/0 conn(0x7f919409de10 legacy=0x7f919406ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:48.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f9198e91700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f9198e91700 1 monclient: found mon.noname-c 2023-09-30T01:54:48.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f9198e91700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f9198e91700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3300/0 2023-09-30T01:54:48.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f9198e91700 1 -- 172.21.15.7:0/3536293837 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919406a380 con 0x7f9194137840 2023-09-30T01:54:48.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7eff7effd700 1 --2- 172.21.15.7:0/2269392554 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7eff84039bf0 0x7eff8403c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7eff7effd700 1 -- 172.21.15.7:0/2269392554 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7eff88047450 con 0x7eff9806e470 2023-09-30T01:54:48.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f2e4faab700 1 Processor -- start 2023-09-30T01:54:48.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f2e4faab700 1 -- start start 2023-09-30T01:54:48.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7f2e4faab700 10 monclient: init 2023-09-30T01:54:48.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7eff967fc700 1 --2- 172.21.15.7:0/2269392554 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7eff84039bf0 0x7eff8403c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.554+0000 7eff967fc700 10 monclient: get_auth_request con 0x7eff84039bf0 auth_method 0 2023-09-30T01:54:48.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4faab700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4faab700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:54:48.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4faab700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e48096a50 0x7f2e4806eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4faab700 10 monclient(hunting): picked mon.noname-a con 0x7f2e48096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:48.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4faab700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e48096e20 0x7f2e480a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4faab700 10 monclient(hunting): picked mon.noname-b con 0x7f2e48096e20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:48.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4faab700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2e48137710 0x7f2e48099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4faab700 10 monclient(hunting): picked mon.noname-c con 0x7f2e48137710 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:48.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4faab700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4faab700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4809c8b0 con 0x7f2e48096a50 2023-09-30T01:54:48.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4faab700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4faab700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4809b7e0 con 0x7f2e48096e20 2023-09-30T01:54:48.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4faab700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4faab700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e48131d90 con 0x7f2e48137710 2023-09-30T01:54:48.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4faab700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4faab700 10 monclient(hunting): authenticate will time out at 1177.620338s 2023-09-30T01:54:48.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4e048700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2e48137710 0x7f2e48099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4d046700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e48096e20 0x7f2e480a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4e048700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2e48137710 0x7f2e48099ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59470/0 (socket says 172.21.15.7:59470) 2023-09-30T01:54:48.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4e048700 1 -- 172.21.15.7:0/3940684532 learned_addr learned my addr 172.21.15.7:0/3940684532 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:48.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4e048700 10 monclient(hunting): get_auth_request con 0x7f2e48137710 auth_method 0 2023-09-30T01:54:48.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4e048700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4e048700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4e048700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4d046700 10 monclient(hunting): get_auth_request con 0x7f2e48096e20 auth_method 0 2023-09-30T01:54:48.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4d046700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4d046700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4d046700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4e048700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4e048700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4e048700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4d046700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4d046700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4d046700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4e048700 10 monclient(hunting): handle_auth_done global_id 4268 payload 824 2023-09-30T01:54:48.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4e048700 1 -- 172.21.15.7:0/3940684532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e48096e20 msgr2=0x7f2e480a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4e048700 1 --2- 172.21.15.7:0/3940684532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e48096e20 0x7f2e480a13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4e048700 1 -- 172.21.15.7:0/3940684532 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e48096a50 msgr2=0x7f2e4806eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4e048700 1 --2- 172.21.15.7:0/3940684532 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e48096a50 0x7f2e4806eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4e048700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4e048700 1 monclient: found mon.noname-c 2023-09-30T01:54:48.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4e048700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4e048700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4e048700 1 -- 172.21.15.7:0/3940684532 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e38009420 con 0x7f2e48137710 2023-09-30T01:54:48.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f2e4d046700 1 --2- 172.21.15.7:0/3940684532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e48096e20 0x7f2e480a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:48.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f15d259c700 1 -- 172.21.15.7:0/3737738535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15d406ebd0 msgr2=0x7f15d409b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.555+0000 7f15d259c700 1 --2- 172.21.15.7:0/3737738535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15d406ebd0 0x7f15d409b910 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f15c400bb90 tx=0x7f15c400b430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c271157212d3583 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f15d159a700 1 -- 172.21.15.7:0/3737738535 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c400dc60 con 0x7f15d406ebd0 2023-09-30T01:54:48.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f15d159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f15d159a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:48.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f15d159a700 10 monclient: dump: 2023-09-30T01:54:48.611 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.611 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.611 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.611 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.611 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.611 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.611 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.612 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.612 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.612 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f15d159a700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f15d159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f15d8de6700 5 monclient: authenticate success, global_id 4263 2023-09-30T01:54:48.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f15d8de6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:48.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f15d159a700 1 -- 172.21.15.7:0/3737738535 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c400ddc0 con 0x7f15d406ebd0 2023-09-30T01:54:48.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f15d159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f15d8de6700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:48.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f15d8de6700 4 set_mon_vals no callback set 2023-09-30T01:54:48.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f15d159a700 1 -- 172.21.15.7:0/3737738535 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c4017640 con 0x7f15d406ebd0 2023-09-30T01:54:48.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f15d159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f15d159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:48.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f15d159a700 10 monclient: dump: 2023-09-30T01:54:48.613 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.613 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.613 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.613 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.613 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.613 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.613 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.613 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.614 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.614 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f9198e91700 1 -- 172.21.15.7:0/3536293837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9194137840 msgr2=0x7f9194139c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f9198e91700 1 --2- 172.21.15.7:0/3536293837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9194137840 0x7f9194139c20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f919406a870 tx=0x7f918c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0945933348a482e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f9192ffd700 1 -- 172.21.15.7:0/3536293837 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918c00e070 con 0x7f9194137840 2023-09-30T01:54:48.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f9192ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f9192ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:48.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f9192ffd700 10 monclient: dump: 2023-09-30T01:54:48.615 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.615 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.615 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.615 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.615 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.615 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.616 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.616 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.616 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.616 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f9192ffd700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f9192ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f919a8f4700 5 monclient: authenticate success, global_id 4261 2023-09-30T01:54:48.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f919a8f4700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:48.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f9192ffd700 1 -- 172.21.15.7:0/3536293837 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918c00bd30 con 0x7f9194137840 2023-09-30T01:54:48.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f9192ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f919a8f4700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:48.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f9192ffd700 1 -- 172.21.15.7:0/3536293837 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918c00bea0 con 0x7f9194137840 2023-09-30T01:54:48.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f9192ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f919a8f4700 4 set_mon_vals no callback set 2023-09-30T01:54:48.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f9192ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:54:48.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.556+0000 7f9192ffd700 10 monclient: dump: 2023-09-30T01:54:48.617 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.617 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.617 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.617 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.617 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.617 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.617 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.617 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.618 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.618 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f15d8de6700 10 monclient: shutdown 2023-09-30T01:54:48.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f15d8de6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f15d8de6700 1 -- 172.21.15.7:0/3737738535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15d406ebd0 msgr2=0x7f15d409b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f15d8de6700 1 --2- 172.21.15.7:0/3737738535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15d406ebd0 0x7f15d409b910 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f15c400bb90 tx=0x7f15c400b430 comp rx=0 tx=0).stop 2023-09-30T01:54:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7eff967fc700 1 --2- 172.21.15.7:0/2269392554 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7eff84039bf0 0x7eff8403c0a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7eff8c006fd0 tx=0x7eff8c006db0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f15d8de6700 1 -- 172.21.15.7:0/3737738535 shutdown_connections 2023-09-30T01:54:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f15d8de6700 1 --2- 172.21.15.7:0/3737738535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15d406ebd0 0x7f15d409b910 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f15d8de6700 1 --2- 172.21.15.7:0/3737738535 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f15d409c7a0 0x7f15d409cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f15d8de6700 1 -- 172.21.15.7:0/3737738535 >> 172.21.15.7:0/3737738535 conn(0x7f15d40680c0 msgr2=0x7f15d40a1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f15d8de6700 1 -- 172.21.15.7:0/3737738535 shutdown_connections 2023-09-30T01:54:48.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f15d8de6700 1 -- 172.21.15.7:0/3737738535 wait complete. 2023-09-30T01:54:48.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f919a8f4700 10 monclient: shutdown 2023-09-30T01:54:48.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f919a8f4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f919a8f4700 1 -- 172.21.15.7:0/3536293837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9194137840 msgr2=0x7f9194139c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f919a8f4700 1 --2- 172.21.15.7:0/3536293837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9194137840 0x7f9194139c20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f919406a870 tx=0x7f918c009780 comp rx=0 tx=0).stop 2023-09-30T01:54:48.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f919a8f4700 1 -- 172.21.15.7:0/3536293837 shutdown_connections 2023-09-30T01:54:48.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f919a8f4700 1 --2- 172.21.15.7:0/3536293837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9194137840 0x7f9194139c20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f919a8f4700 1 --2- 172.21.15.7:0/3536293837 >> v2:172.21.15.7:3300/0 conn(0x7f919409e1e0 0x7f919409e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f919a8f4700 1 -- 172.21.15.7:0/3536293837 >> 172.21.15.7:0/3536293837 conn(0x7f91940680c0 msgr2=0x7f919409ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f919a8f4700 1 -- 172.21.15.7:0/3536293837 shutdown_connections 2023-09-30T01:54:48.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f919a8f4700 1 -- 172.21.15.7:0/3536293837 wait complete. 2023-09-30T01:54:48.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f15d8de6700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.557+0000 7f15d8de6700 10 monclient: monmap: 2023-09-30T01:54:48.622 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.622 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:48.622 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:48.622 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:48.622 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.622 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.622 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:48.622 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:48.623 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:48.623 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f2e4e048700 1 -- 172.21.15.7:0/3940684532 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2e48137710 msgr2=0x7f2e48099ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f2e4e048700 1 --2- 172.21.15.7:0/3940684532 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2e48137710 0x7f2e48099ea0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2e4000ccc0 tx=0x7f2e40005400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d8de6700 1 Processor -- start 2023-09-30T01:54:48.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d8de6700 1 -- start start 2023-09-30T01:54:48.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d8de6700 10 monclient: init 2023-09-30T01:54:48.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f919a8f4700 10 monclient: build_initial_monmap 2023-09-30T01:54:48.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f919a8f4700 10 monclient: monmap: 2023-09-30T01:54:48.624 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:48.625 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f919a8f4700 1 Processor -- start 2023-09-30T01:54:48.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d8de6700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d8de6700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:54:48.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d8de6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15d406ebd0 0x7f15d4141d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d8de6700 10 monclient(hunting): picked mon.noname-a con 0x7f15d406ebd0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:48.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d8de6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15d409c7a0 0x7f15d41309a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d8de6700 10 monclient(hunting): picked mon.noname-b con 0x7f15d409c7a0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:48.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d8de6700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f15d409d0e0 0x7f15d411e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d8de6700 10 monclient(hunting): picked mon.noname-c con 0x7f15d409d0e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:48.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d8de6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d8de6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d41220c0 con 0x7f15d406ebd0 2023-09-30T01:54:48.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d8de6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d8de6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d409d9e0 con 0x7f15d409c7a0 2023-09-30T01:54:48.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d8de6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d8de6700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d4121de0 con 0x7f15d409d0e0 2023-09-30T01:54:48.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d8de6700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d8de6700 10 monclient(hunting): authenticate will time out at 1177.623736s 2023-09-30T01:54:48.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f919a8f4700 1 -- start start 2023-09-30T01:54:48.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f919a8f4700 10 monclient: init 2023-09-30T01:54:48.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15d406ebd0 0x7f15d4141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d2d9d700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f15d409d0e0 0x7f15d411e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d259c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15d406ebd0 0x7f15d4141d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54836/0 (socket says 172.21.15.7:54836) 2023-09-30T01:54:48.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d259c700 1 -- 172.21.15.7:0/3763788317 learned_addr learned my addr 172.21.15.7:0/3763788317 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:48.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d259c700 10 monclient(hunting): get_auth_request con 0x7f15d406ebd0 auth_method 0 2023-09-30T01:54:48.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d2d9d700 10 monclient(hunting): get_auth_request con 0x7f15d409d0e0 auth_method 0 2023-09-30T01:54:48.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d2d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d2d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d1d9b700 1 --2- 172.21.15.7:0/3763788317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15d409c7a0 0x7f15d41309a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d1d9b700 10 monclient(hunting): get_auth_request con 0x7f15d409c7a0 auth_method 0 2023-09-30T01:54:48.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d1d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d1d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.558+0000 7f15d259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f15d2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f15d2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f15d2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f15d259c700 10 monclient(hunting): handle_auth_done global_id 4269 payload 824 2023-09-30T01:54:48.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f15d259c700 1 -- 172.21.15.7:0/3763788317 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f15d409d0e0 msgr2=0x7f15d411e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f15d259c700 1 --2- 172.21.15.7:0/3763788317 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f15d409d0e0 0x7f15d411e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f15d259c700 1 -- 172.21.15.7:0/3763788317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15d409c7a0 msgr2=0x7f15d41309a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f15d259c700 1 --2- 172.21.15.7:0/3763788317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15d409c7a0 0x7f15d41309a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f15d259c700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f15d259c700 1 monclient: found mon.noname-a 2023-09-30T01:54:48.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f15d259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f15d259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:48.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f15d259c700 1 -- 172.21.15.7:0/3763788317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c400b6d0 con 0x7f15d406ebd0 2023-09-30T01:54:48.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f15d2d9d700 1 --2- 172.21.15.7:0/3763788317 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f15d409d0e0 0x7f15d411e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:48.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f15d1d9b700 1 --2- 172.21.15.7:0/3763788317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15d409c7a0 0x7f15d41309a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:54:48.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f919a8f4700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:48.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f919a8f4700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:54:48.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f919a8f4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f919409de10 0x7f9194141cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f919a8f4700 10 monclient(hunting): picked mon.noname-a con 0x7f919409de10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:48.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f919a8f4700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f919409e1e0 0x7f91941421f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f919a8f4700 10 monclient(hunting): picked mon.noname-c con 0x7f919409e1e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:48.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f919a8f4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9194137840 0x7f9194142730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f919a8f4700 10 monclient(hunting): picked mon.noname-b con 0x7f9194137840 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:48.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f919a8f4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f919a8f4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91941362e0 con 0x7f919409de10 2023-09-30T01:54:48.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f919a8f4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f919a8f4700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9194133610 con 0x7f9194137840 2023-09-30T01:54:48.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f919a8f4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:48.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f919a8f4700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f919409a800 con 0x7f919409e1e0 2023-09-30T01:54:48.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f919a8f4700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:48.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f919a8f4700 10 monclient(hunting): authenticate will time out at 1177.624510s 2023-09-30T01:54:48.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f91937fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f919409e1e0 0x7f91941421f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f91937fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f919409e1e0 0x7f91941421f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59476/0 (socket says 172.21.15.7:59476) 2023-09-30T01:54:48.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f91937fe700 1 -- 172.21.15.7:0/1130955286 learned_addr learned my addr 172.21.15.7:0/1130955286 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:48.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f91937fe700 10 monclient(hunting): get_auth_request con 0x7f919409e1e0 auth_method 0 2023-09-30T01:54:48.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f91937fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f91937fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f91937fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f91937fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f91937fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f9198e91700 1 --2- 172.21.15.7:0/1130955286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9194137840 0x7f9194142730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f91937fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f9198e91700 10 monclient(hunting): get_auth_request con 0x7f9194137840 auth_method 0 2023-09-30T01:54:48.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f9198e91700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:48.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f9198e91700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:48.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f9198e91700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:48.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f9198e91700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:48.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f9198e91700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:48.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f9198e91700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:48.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f91937fe700 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2023-09-30T01:54:48.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f91937fe700 1 -- 172.21.15.7:0/1130955286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9194137840 msgr2=0x7f9194142730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f91937fe700 1 --2- 172.21.15.7:0/1130955286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9194137840 0x7f9194142730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f91937fe700 1 -- 172.21.15.7:0/1130955286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f919409de10 msgr2=0x7f9194141cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f91937fe700 1 --2- 172.21.15.7:0/1130955286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f919409de10 0x7f9194141cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f91937fe700 10 monclient: _finish_hunting 0 2023-09-30T01:54:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f91937fe700 1 monclient: found mon.noname-c 2023-09-30T01:54:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f91937fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f91937fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.559+0000 7f91937fe700 1 -- 172.21.15.7:0/1130955286 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918c009a20 con 0x7f919409e1e0 2023-09-30T01:54:48.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7f2e3dffb700 1 -- 172.21.15.7:0/3940684532 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4000ab40 con 0x7f2e48137710 2023-09-30T01:54:48.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7f2e3dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7f2e3dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:48.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7f2e3dffb700 10 monclient: dump: 2023-09-30T01:54:48.640 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.640 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.640 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.640 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.640 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.640 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.641 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.641 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.641 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.641 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7f2e3dffb700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7f2e3dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7f2e3dffb700 1 -- 172.21.15.7:0/3940684532 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4000acc0 con 0x7f2e48137710 2023-09-30T01:54:48.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7f2e3dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7f2e3dffb700 1 -- 172.21.15.7:0/3940684532 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e40007340 con 0x7f2e48137710 2023-09-30T01:54:48.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7f2e4faab700 5 monclient: authenticate success, global_id 4268 2023-09-30T01:54:48.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7f2e4faab700 10 monclient: _renew_subs 2023-09-30T01:54:48.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7f2e4faab700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7f2e4faab700 1 -- 172.21.15.7:0/3940684532 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e4809c5d0 con 0x7f2e48137710 2023-09-30T01:54:48.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7f2e3ffff700 4 set_mon_vals no callback set 2023-09-30T01:54:48.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7f2e3dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7f2e3dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:48.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7f2e3dffb700 10 monclient: dump: 2023-09-30T01:54:48.642 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.642 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.642 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.642 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.642 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.642 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.643 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.643 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.643 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.643 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7eff7effd700 1 -- 172.21.15.7:0/2269392554 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7eff88018260 con 0x7eff9806e470 2023-09-30T01:54:48.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7eff7effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:48.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7eff7effd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:48.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7f2e4faab700 10 monclient: _renew_subs 2023-09-30T01:54:48.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7f2e4faab700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.560+0000 7f2e4faab700 1 -- 172.21.15.7:0/3940684532 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e48146380 con 0x7f2e48137710 2023-09-30T01:54:48.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f15d259c700 1 -- 172.21.15.7:0/3763788317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15d406ebd0 msgr2=0x7f15d4141d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f15d259c700 1 --2- 172.21.15.7:0/3763788317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15d406ebd0 0x7f15d4141d40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f15d406fb70 tx=0x7f15c4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f2e3dffb700 1 -- 172.21.15.7:0/3940684532 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f2e400076f0 con 0x7f2e48137710 2023-09-30T01:54:48.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f15c27fc700 1 -- 172.21.15.7:0/3763788317 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c4006af0 con 0x7f15d406ebd0 2023-09-30T01:54:48.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f15c27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f15c27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:48.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f15c27fc700 10 monclient: dump: 2023-09-30T01:54:48.645 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.646 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.646 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.646 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.646 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.646 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.646 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.646 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.646 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.646 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f15c27fc700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f15c27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f15c27fc700 1 -- 172.21.15.7:0/3763788317 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c401f410 con 0x7f15d406ebd0 2023-09-30T01:54:48.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f15c27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f15c27fc700 1 -- 172.21.15.7:0/3763788317 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c401f7d0 con 0x7f15d406ebd0 2023-09-30T01:54:48.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f2e4faab700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:48.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f2e4faab700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:48.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f2e4faab700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f15c27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f15c27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:48.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f15c27fc700 10 monclient: dump: 2023-09-30T01:54:48.648 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.648 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.648 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.648 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.648 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.648 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.648 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.648 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.649 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.649 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f2e3dffb700 1 --2- 172.21.15.7:0/3940684532 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2e2c0357e0 0x7f2e2c037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f15d8de6700 5 monclient: authenticate success, global_id 4269 2023-09-30T01:54:48.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f15d8de6700 10 monclient: _renew_subs 2023-09-30T01:54:48.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f15d8de6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:48.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f15d8de6700 1 -- 172.21.15.7:0/3763788317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d4120c10 con 0x7f15d406ebd0 2023-09-30T01:54:48.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f15d0d99700 4 set_mon_vals no callback set 2023-09-30T01:54:48.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f2e4faab700 1 -- 172.21.15.7:0/3940684532 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e48064c10 con 0x7f2e48137710 2023-09-30T01:54:48.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f2e3dffb700 1 -- 172.21.15.7:0/3940684532 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f2e40048170 con 0x7f2e48137710 2023-09-30T01:54:48.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.561+0000 7f15d8de6700 10 monclient: _renew_subs 2023-09-30T01:54:48.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.562+0000 7f15d8de6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:48.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.562+0000 7f15d8de6700 1 -- 172.21.15.7:0/3763788317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d411ffd0 con 0x7f15d406ebd0 2023-09-30T01:54:48.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.562+0000 7f2e4d847700 1 --2- 172.21.15.7:0/3940684532 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2e2c0357e0 0x7f2e2c037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.562+0000 7f2e4d847700 10 monclient: get_auth_request con 0x7f2e2c0357e0 auth_method 0 2023-09-30T01:54:48.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.562+0000 7f15d8de6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:48.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.562+0000 7f15d8de6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:48.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.562+0000 7f15d8de6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:48.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.562+0000 7f15d8de6700 1 -- 172.21.15.7:0/3763788317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d4064c10 con 0x7f15d406ebd0 2023-09-30T01:54:48.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.563+0000 7f15c27fc700 1 -- 172.21.15.7:0/3763788317 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f15c4015070 con 0x7f15d406ebd0 2023-09-30T01:54:48.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.564+0000 7f15c27fc700 1 --2- 172.21.15.7:0/3763788317 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f15b4035830 0x7f15b4037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.564+0000 7f15c27fc700 1 -- 172.21.15.7:0/3763788317 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f15c4048c30 con 0x7f15d406ebd0 2023-09-30T01:54:48.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.564+0000 7f15d1d9b700 1 --2- 172.21.15.7:0/3763788317 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f15b4035830 0x7f15b4037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.564+0000 7f15d1d9b700 10 monclient: get_auth_request con 0x7f15b4035830 auth_method 0 2023-09-30T01:54:48.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.564+0000 7f2e4d847700 1 --2- 172.21.15.7:0/3940684532 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2e2c0357e0 0x7f2e2c037c90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2e44007fd0 tx=0x7f2e44007b20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.565+0000 7f91937fe700 1 -- 172.21.15.7:0/1130955286 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f919409e1e0 msgr2=0x7f91941421f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:48.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.565+0000 7f91937fe700 1 --2- 172.21.15.7:0/1130955286 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f919409e1e0 0x7f91941421f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f918800ea80 tx=0x7f9188009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.566+0000 7f15d1d9b700 1 --2- 172.21.15.7:0/3763788317 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f15b4035830 0x7f15b4037ce0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f15d411fa30 tx=0x7f15c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.566+0000 7f917bfff700 1 -- 172.21.15.7:0/1130955286 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9188009640 con 0x7f919409e1e0 2023-09-30T01:54:48.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f917bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f917bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:48.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f917bfff700 10 monclient: dump: 2023-09-30T01:54:48.655 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.656 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.656 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.656 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.656 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.656 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.656 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.656 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.656 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.656 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f917bfff700 10 monclient: _finish_auth 0 2023-09-30T01:54:48.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f917bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:48.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f917bfff700 1 -- 172.21.15.7:0/1130955286 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918800ca60 con 0x7f919409e1e0 2023-09-30T01:54:48.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f919a8f4700 5 monclient: authenticate success, global_id 4274 2023-09-30T01:54:48.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f919a8f4700 10 monclient: _renew_subs 2023-09-30T01:54:48.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f919a8f4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f919a8f4700 1 -- 172.21.15.7:0/1130955286 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9194136930 con 0x7f919409e1e0 2023-09-30T01:54:48.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f919a8f4700 10 monclient: _renew_subs 2023-09-30T01:54:48.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f919a8f4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f919a8f4700 1 -- 172.21.15.7:0/1130955286 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f919411df50 con 0x7f919409e1e0 2023-09-30T01:54:48.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f917bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:48.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f2e3dffb700 1 -- 172.21.15.7:0/3940684532 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2e40010d60 con 0x7f2e48137710 2023-09-30T01:54:48.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f2e3dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:48.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f2e3dffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:48.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f917bfff700 1 -- 172.21.15.7:0/1130955286 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918800cde0 con 0x7f919409e1e0 2023-09-30T01:54:48.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f917bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:48.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f917bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:48.659 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f917bfff700 10 monclient: dump: 2023-09-30T01:54:48.659 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:48.659 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:48.659 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.660 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:48.660 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:48.660 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:48.660 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:48.660 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:48.660 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:48.660 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:48.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.567+0000 7f91927fc700 4 set_mon_vals no callback set 2023-09-30T01:54:48.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.568+0000 7f917bfff700 1 -- 172.21.15.7:0/1130955286 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f9188013680 con 0x7f919409e1e0 2023-09-30T01:54:48.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.568+0000 7f919a8f4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:48.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.568+0000 7f919a8f4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:48.660 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.568+0000 7f919a8f4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.568+0000 7f919a8f4700 1 -- 172.21.15.7:0/1130955286 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91740050a0 con 0x7f919409e1e0 2023-09-30T01:54:48.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.568+0000 7f917bfff700 1 --2- 172.21.15.7:0/1130955286 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9180035830 0x7f9180037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:48.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.568+0000 7f917bfff700 1 -- 172.21.15.7:0/1130955286 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f9188048100 con 0x7f919409e1e0 2023-09-30T01:54:48.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.568+0000 7f9193fff700 1 --2- 172.21.15.7:0/1130955286 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9180035830 0x7f9180037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:48.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.568+0000 7f15c27fc700 1 -- 172.21.15.7:0/3763788317 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f15c401e370 con 0x7f15d406ebd0 2023-09-30T01:54:48.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.568+0000 7f15c27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:48.661 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.568+0000 7f15c27fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:48.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.568+0000 7f9193fff700 10 monclient: get_auth_request con 0x7f9180035830 auth_method 0 2023-09-30T01:54:48.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.571+0000 7f9193fff700 1 --2- 172.21.15.7:0/1130955286 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9180035830 0x7f9180037ce0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9184007fd0 tx=0x7f9184007b20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:48.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.577+0000 7f917bfff700 1 -- 172.21.15.7:0/1130955286 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9188013930 con 0x7f919409e1e0 2023-09-30T01:54:48.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.577+0000 7f917bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:48.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.577+0000 7f917bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:48.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.737+0000 7f47ac565700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T01:54:48.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.737+0000 7f47ac565700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T01:54:48.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.737+0000 7f47ac565700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.50:3300/0 2023-09-30T01:54:48.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.737+0000 7f47ac565700 1 -- 172.21.15.7:0/762937196 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f47a409cfe0 con 0x7f47a409dd00 2023-09-30T01:54:48.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.738+0000 7f479a7fc700 1 -- 172.21.15.7:0/762937196 <== mon.1 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4794019360 con 0x7f47a409dd00 2023-09-30T01:54:48.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.738+0000 7f479a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T01:54:48.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.738+0000 7f479a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:48.739 INFO:teuthology.orchestra.run.smithi007.stdout:47244640258 2023-09-30T01:54:48.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.741+0000 7f47ac565700 1 -- 172.21.15.7:0/762937196 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f4788035830 msgr2=0x7f4788037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.741+0000 7f47ac565700 1 --2- 172.21.15.7:0/762937196 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f4788035830 0x7f4788037ce0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f47a40a0e30 tx=0x7f47a0017000 comp rx=0 tx=0).stop 2023-09-30T01:54:48.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.741+0000 7f47ac565700 10 monclient: shutdown 2023-09-30T01:54:48.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.741+0000 7f47ac565700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.741+0000 7f47ac565700 1 -- 172.21.15.7:0/762937196 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47a409dd00 msgr2=0x7f47a413b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.741+0000 7f47ac565700 1 --2- 172.21.15.7:0/762937196 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47a409dd00 0x7f47a413b830 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f479400b5b0 tx=0x7f47940090f0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.741+0000 7f47ac565700 1 -- 172.21.15.7:0/762937196 shutdown_connections 2023-09-30T01:54:48.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.741+0000 7f47ac565700 1 --2- 172.21.15.7:0/762937196 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f4788035830 0x7f4788037ce0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.741+0000 7f47ac565700 1 --2- 172.21.15.7:0/762937196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47a409d250 0x7f47a413c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.741+0000 7f47ac565700 1 --2- 172.21.15.7:0/762937196 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f47a409e660 0x7f47a413f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.741+0000 7f47ac565700 1 --2- 172.21.15.7:0/762937196 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f47a409dd00 0x7f47a413b830 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.741+0000 7f47ac565700 1 -- 172.21.15.7:0/762937196 >> 172.21.15.7:0/762937196 conn(0x7f47a4071910 msgr2=0x7f47a40a1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.741+0000 7f47ac565700 1 -- 172.21.15.7:0/762937196 shutdown_connections 2023-09-30T01:54:48.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.741+0000 7f47ac565700 1 -- 172.21.15.7:0/762937196 wait complete. 2023-09-30T01:54:48.758 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.2 2023-09-30T01:54:48.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.766+0000 7eff9da47700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T01:54:48.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.766+0000 7eff9da47700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T01:54:48.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.766+0000 7eff9da47700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.766+0000 7eff9da47700 1 -- 172.21.15.7:0/2269392554 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7eff980a1240 con 0x7eff9806e470 2023-09-30T01:54:48.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.766+0000 7eff7effd700 1 -- 172.21.15.7:0/2269392554 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7eff88017020 con 0x7eff9806e470 2023-09-30T01:54:48.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.766+0000 7eff7effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T01:54:48.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.766+0000 7eff7effd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:48.767 INFO:teuthology.orchestra.run.smithi007.stdout:47244640258 2023-09-30T01:54:48.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.768+0000 7eff9da47700 1 -- 172.21.15.7:0/2269392554 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7eff84039bf0 msgr2=0x7eff8403c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.768+0000 7eff9da47700 1 --2- 172.21.15.7:0/2269392554 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7eff84039bf0 0x7eff8403c0a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7eff8c006fd0 tx=0x7eff8c006db0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.768+0000 7eff9da47700 10 monclient: shutdown 2023-09-30T01:54:48.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.768+0000 7eff9da47700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.768+0000 7eff9da47700 1 -- 172.21.15.7:0/2269392554 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7eff9806e470 msgr2=0x7eff9813d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.768+0000 7eff9da47700 1 --2- 172.21.15.7:0/2269392554 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7eff9806e470 0x7eff9813d530 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7eff8800c5e0 tx=0x7eff88004a80 comp rx=0 tx=0).stop 2023-09-30T01:54:48.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.768+0000 7eff9da47700 1 -- 172.21.15.7:0/2269392554 shutdown_connections 2023-09-30T01:54:48.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.768+0000 7eff9da47700 1 --2- 172.21.15.7:0/2269392554 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7eff84039bf0 0x7eff8403c0a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.768+0000 7eff9da47700 1 --2- 172.21.15.7:0/2269392554 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff98070cf0 0x7eff9812d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.769+0000 7eff9da47700 1 --2- 172.21.15.7:0/2269392554 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7eff9806e470 0x7eff9813d530 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.769+0000 7eff9da47700 1 --2- 172.21.15.7:0/2269392554 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7eff9806e840 0x7eff9812d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.769+0000 7eff9da47700 1 -- 172.21.15.7:0/2269392554 >> 172.21.15.7:0/2269392554 conn(0x7eff98067f20 msgr2=0x7eff9812e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.769+0000 7eff9da47700 1 -- 172.21.15.7:0/2269392554 shutdown_connections 2023-09-30T01:54:48.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.769+0000 7eff9da47700 1 -- 172.21.15.7:0/2269392554 wait complete. 2023-09-30T01:54:48.781 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.1 2023-09-30T01:54:48.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.789+0000 7f919a8f4700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T01:54:48.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.789+0000 7f919a8f4700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T01:54:48.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.789+0000 7f919a8f4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.789+0000 7f919a8f4700 1 -- 172.21.15.7:0/1130955286 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f9174005350 con 0x7f919409e1e0 2023-09-30T01:54:48.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.789+0000 7f917bfff700 1 -- 172.21.15.7:0/1130955286 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9188021020 con 0x7f919409e1e0 2023-09-30T01:54:48.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.789+0000 7f917bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T01:54:48.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.789+0000 7f917bfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:48.790 INFO:teuthology.orchestra.run.smithi007.stdout:47244640257 2023-09-30T01:54:48.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.791+0000 7f919a8f4700 1 -- 172.21.15.7:0/1130955286 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9180035830 msgr2=0x7f9180037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.791+0000 7f919a8f4700 1 --2- 172.21.15.7:0/1130955286 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9180035830 0x7f9180037ce0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9184007fd0 tx=0x7f9184007b20 comp rx=0 tx=0).stop 2023-09-30T01:54:48.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.791+0000 7f919a8f4700 10 monclient: shutdown 2023-09-30T01:54:48.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.791+0000 7f919a8f4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.791+0000 7f919a8f4700 1 -- 172.21.15.7:0/1130955286 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f919409e1e0 msgr2=0x7f91941421f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.791+0000 7f919a8f4700 1 --2- 172.21.15.7:0/1130955286 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f919409e1e0 0x7f91941421f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f918800ea80 tx=0x7f9188009250 comp rx=0 tx=0).stop 2023-09-30T01:54:48.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.791+0000 7f919a8f4700 1 -- 172.21.15.7:0/1130955286 shutdown_connections 2023-09-30T01:54:48.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.791+0000 7f919a8f4700 1 --2- 172.21.15.7:0/1130955286 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9180035830 0x7f9180037ce0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.791+0000 7f919a8f4700 1 --2- 172.21.15.7:0/1130955286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f919409de10 0x7f9194141cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.791+0000 7f919a8f4700 1 --2- 172.21.15.7:0/1130955286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9194137840 0x7f9194142730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.791+0000 7f919a8f4700 1 --2- 172.21.15.7:0/1130955286 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f919409e1e0 0x7f91941421f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.791+0000 7f919a8f4700 1 -- 172.21.15.7:0/1130955286 >> 172.21.15.7:0/1130955286 conn(0x7f91940680c0 msgr2=0x7f919409c660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.791+0000 7f919a8f4700 1 -- 172.21.15.7:0/1130955286 shutdown_connections 2023-09-30T01:54:48.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.791+0000 7f919a8f4700 1 -- 172.21.15.7:0/1130955286 wait complete. 2023-09-30T01:54:48.803 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640257 for osd.0 2023-09-30T01:54:48.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.879+0000 7f15d8de6700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T01:54:48.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.879+0000 7f15d8de6700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T01:54:48.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.879+0000 7f15d8de6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:48.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.879+0000 7f15d8de6700 1 -- 172.21.15.7:0/3763788317 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f15d4142be0 con 0x7f15d406ebd0 2023-09-30T01:54:48.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.880+0000 7f15c27fc700 1 -- 172.21.15.7:0/3763788317 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f15c4024020 con 0x7f15d406ebd0 2023-09-30T01:54:48.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.880+0000 7f15c27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T01:54:48.880 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.880+0000 7f15c27fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:48.880 INFO:teuthology.orchestra.run.smithi007.stdout:47244640257 2023-09-30T01:54:48.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.880+0000 7f2e4faab700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T01:54:48.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.880+0000 7f2e4faab700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T01:54:48.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.880+0000 7f2e4faab700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:48.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.880+0000 7f2e4faab700 1 -- 172.21.15.7:0/3940684532 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f2e4809caf0 con 0x7f2e48137710 2023-09-30T01:54:48.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.881+0000 7f2e3dffb700 1 -- 172.21.15.7:0/3940684532 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2e40029090 con 0x7f2e48137710 2023-09-30T01:54:48.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.881+0000 7f2e3dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T01:54:48.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.881+0000 7f2e3dffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:48.882 INFO:teuthology.orchestra.run.smithi007.stdout:47244640258 2023-09-30T01:54:48.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.882+0000 7f15d8de6700 1 -- 172.21.15.7:0/3763788317 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f15b4035830 msgr2=0x7f15b4037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.882+0000 7f15d8de6700 1 --2- 172.21.15.7:0/3763788317 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f15b4035830 0x7f15b4037ce0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f15d411fa30 tx=0x7f15c8008040 comp rx=0 tx=0).stop 2023-09-30T01:54:48.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.882+0000 7f15d8de6700 10 monclient: shutdown 2023-09-30T01:54:48.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.882+0000 7f15d8de6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.882+0000 7f15d8de6700 1 -- 172.21.15.7:0/3763788317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15d406ebd0 msgr2=0x7f15d4141d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.882+0000 7f15d8de6700 1 --2- 172.21.15.7:0/3763788317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15d406ebd0 0x7f15d4141d40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f15d406fb70 tx=0x7f15c4019000 comp rx=0 tx=0).stop 2023-09-30T01:54:48.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.882+0000 7f15d8de6700 1 -- 172.21.15.7:0/3763788317 shutdown_connections 2023-09-30T01:54:48.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.882+0000 7f15d8de6700 1 --2- 172.21.15.7:0/3763788317 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f15b4035830 0x7f15b4037ce0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.882+0000 7f15d8de6700 1 --2- 172.21.15.7:0/3763788317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f15d406ebd0 0x7f15d4141d40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.882+0000 7f15d8de6700 1 --2- 172.21.15.7:0/3763788317 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f15d409d0e0 0x7f15d411e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.882+0000 7f15d8de6700 1 --2- 172.21.15.7:0/3763788317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f15d409c7a0 0x7f15d41309a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.882+0000 7f15d8de6700 1 -- 172.21.15.7:0/3763788317 >> 172.21.15.7:0/3763788317 conn(0x7f15d40680c0 msgr2=0x7f15d41317b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.882+0000 7f15d8de6700 1 -- 172.21.15.7:0/3763788317 shutdown_connections 2023-09-30T01:54:48.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.882+0000 7f15d8de6700 1 -- 172.21.15.7:0/3763788317 wait complete. 2023-09-30T01:54:48.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.883+0000 7f2e4faab700 1 -- 172.21.15.7:0/3940684532 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2e2c0357e0 msgr2=0x7f2e2c037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.883+0000 7f2e4faab700 1 --2- 172.21.15.7:0/3940684532 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2e2c0357e0 0x7f2e2c037c90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2e44007fd0 tx=0x7f2e44007b20 comp rx=0 tx=0).stop 2023-09-30T01:54:48.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.883+0000 7f2e4faab700 10 monclient: shutdown 2023-09-30T01:54:48.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.883+0000 7f2e4faab700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:48.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.883+0000 7f2e4faab700 1 -- 172.21.15.7:0/3940684532 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2e48137710 msgr2=0x7f2e48099ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:48.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.883+0000 7f2e4faab700 1 --2- 172.21.15.7:0/3940684532 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2e48137710 0x7f2e48099ea0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2e4000ccc0 tx=0x7f2e40005400 comp rx=0 tx=0).stop 2023-09-30T01:54:48.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.883+0000 7f2e4faab700 1 -- 172.21.15.7:0/3940684532 shutdown_connections 2023-09-30T01:54:48.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.883+0000 7f2e4faab700 1 --2- 172.21.15.7:0/3940684532 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2e2c0357e0 0x7f2e2c037c90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.883+0000 7f2e4faab700 1 --2- 172.21.15.7:0/3940684532 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2e48096a50 0x7f2e4806eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.883+0000 7f2e4faab700 1 --2- 172.21.15.7:0/3940684532 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2e48137710 0x7f2e48099ea0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.883+0000 7f2e4faab700 1 --2- 172.21.15.7:0/3940684532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e48096e20 0x7f2e480a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:48.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.883+0000 7f2e4faab700 1 -- 172.21.15.7:0/3940684532 >> 172.21.15.7:0/3940684532 conn(0x7f2e48070cf0 msgr2=0x7f2e4812f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:48.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.883+0000 7f2e4faab700 1 -- 172.21.15.7:0/3940684532 shutdown_connections 2023-09-30T01:54:48.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:48.883+0000 7f2e4faab700 1 -- 172.21.15.7:0/3940684532 wait complete. 2023-09-30T01:54:48.896 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640257 for osd.3 2023-09-30T01:54:48.896 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.5 2023-09-30T01:54:49.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.126+0000 7ffa70ff9700 1 -- 172.21.15.7:0/2299502451 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ffa6800d340 con 0x7ffa74096a50 2023-09-30T01:54:49.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.126+0000 7ffa70ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:49.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.126+0000 7ffa70ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:49.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.319+0000 7ffa7b08f700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T01:54:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.319+0000 7ffa7b08f700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T01:54:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.319+0000 7ffa7b08f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.319+0000 7ffa7b08f700 1 -- 172.21.15.7:0/2299502451 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ffa74097770 con 0x7ffa74096a50 2023-09-30T01:54:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.319+0000 7ffa70ff9700 1 -- 172.21.15.7:0/2299502451 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ffa68028090 con 0x7ffa74096a50 2023-09-30T01:54:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.319+0000 7ffa70ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T01:54:49.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.319+0000 7ffa70ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:49.320 INFO:teuthology.orchestra.run.smithi007.stdout:47244640258 2023-09-30T01:54:49.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.321+0000 7ffa7b08f700 1 -- 172.21.15.7:0/2299502451 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ffa58035830 msgr2=0x7ffa58037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:49.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.321+0000 7ffa7b08f700 1 --2- 172.21.15.7:0/2299502451 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ffa58035830 0x7ffa58037ce0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ffa7409f2a0 tx=0x7ffa60016000 comp rx=0 tx=0).stop 2023-09-30T01:54:49.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.321+0000 7ffa7b08f700 10 monclient: shutdown 2023-09-30T01:54:49.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.321+0000 7ffa7b08f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:49.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.321+0000 7ffa7b08f700 1 -- 172.21.15.7:0/2299502451 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa74096a50 msgr2=0x7ffa7412df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:49.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.321+0000 7ffa7b08f700 1 --2- 172.21.15.7:0/2299502451 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa74096a50 0x7ffa7412df80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ffa6800b870 tx=0x7ffa68009250 comp rx=0 tx=0).stop 2023-09-30T01:54:49.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.321+0000 7ffa7b08f700 1 -- 172.21.15.7:0/2299502451 shutdown_connections 2023-09-30T01:54:49.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.321+0000 7ffa7b08f700 1 --2- 172.21.15.7:0/2299502451 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ffa58035830 0x7ffa58037ce0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:49.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.321+0000 7ffa7b08f700 1 --2- 172.21.15.7:0/2299502451 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa7412ef00 0x7ffa7409dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:49.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.321+0000 7ffa7b08f700 1 --2- 172.21.15.7:0/2299502451 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ffa74096a50 0x7ffa7412df80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:49.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.321+0000 7ffa7b08f700 1 --2- 172.21.15.7:0/2299502451 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa74096e20 0x7ffa7409da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:49.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.321+0000 7ffa7b08f700 1 -- 172.21.15.7:0/2299502451 >> 172.21.15.7:0/2299502451 conn(0x7ffa74070cf0 msgr2=0x7ffa74131110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:49.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.322+0000 7ffa7b08f700 1 -- 172.21.15.7:0/2299502451 shutdown_connections 2023-09-30T01:54:49.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.322+0000 7ffa7b08f700 1 -- 172.21.15.7:0/2299502451 wait complete. 2023-09-30T01:54:49.334 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.4 2023-09-30T01:54:49.334 DEBUG:teuthology.parallel:result is None 2023-09-30T01:54:49.759 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T01:54:49.782 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T01:54:49.804 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T01:54:49.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.888+0000 7f7361c74700 10 monclient: get_monmap_and_config 2023-09-30T01:54:49.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.888+0000 7f7361c74700 10 monclient: build_initial_monmap 2023-09-30T01:54:49.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.888+0000 7f7361c74700 10 monclient: monmap: 2023-09-30T01:54:49.889 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:49.889 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:49.889 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:49.889596+0000 2023-09-30T01:54:49.890 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:49.889596+0000 2023-09-30T01:54:49.890 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:49.890 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:49.890 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:49.890 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:49.891 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:49.891 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:49.891 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:49.891 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:49.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.889+0000 7f7361c74700 1 Processor -- start 2023-09-30T01:54:49.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.889+0000 7f7361c74700 1 -- start start 2023-09-30T01:54:49.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.889+0000 7f7361c74700 10 monclient: init 2023-09-30T01:54:49.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.889+0000 7f7361c74700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:49.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.889+0000 7f7361c74700 10 monclient: _add_conns ranks=[4,0,1,3,2] 2023-09-30T01:54:49.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.889+0000 7f7361c74700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f735c096a50 0x7f735c132680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:49.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.889+0000 7f7361c74700 10 monclient(hunting): picked mon.noname-c con 0x7f735c096a50 addr v2:172.21.15.50:3300/0 2023-09-30T01:54:49.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.889+0000 7f7361c74700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f735c096e20 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:49.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.889+0000 7f7361c74700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f735c137710 addr v1:172.21.15.50:6789/0 2023-09-30T01:54:49.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.889+0000 7f7361c74700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f735c099440 con 0x7f735c096e20 2023-09-30T01:54:49.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.889+0000 7f7361c74700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f735c097b40 con 0x7f735c137710 2023-09-30T01:54:49.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.889+0000 7f7361c74700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:49.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.889+0000 7f7361c74700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f735c099160 con 0x7f735c096a50 2023-09-30T01:54:49.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.889+0000 7f7361c74700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:49.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.889+0000 7f7361c74700 10 monclient(hunting): authenticate will time out at 1178.954919s 2023-09-30T01:54:49.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.890+0000 7f735affd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f735c096e20 0x7f735c097220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58124/0 (socket says 172.21.15.7:58124) 2023-09-30T01:54:49.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.890+0000 7f735affd700 1 -- 172.21.15.7:0/3154044661 learned_addr learned my addr 172.21.15.7:0/3154044661 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:54:49.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.890+0000 7f735b7fe700 1 --2- 172.21.15.7:0/3154044661 >> v2:172.21.15.50:3300/0 conn(0x7f735c096a50 0x7f735c132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:49.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.890+0000 7f735b7fe700 10 monclient(hunting): get_auth_request con 0x7f735c096a50 auth_method 0 2023-09-30T01:54:49.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.890+0000 7f735b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:49.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.890+0000 7f735b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:49.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.890+0000 7f735b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:49.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.890+0000 7f735b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:49.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.890+0000 7f735b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:49.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.890+0000 7f735b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:49.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.890+0000 7f735b7fe700 10 monclient(hunting): handle_auth_done global_id 4273 payload 274 2023-09-30T01:54:49.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.890+0000 7f735b7fe700 1 -- 172.21.15.7:0/3154044661 >> v1:172.21.15.50:6789/0 conn(0x7f735c137710 legacy=0x7f735c139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:49.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.890+0000 7f735b7fe700 1 -- 172.21.15.7:0/3154044661 >> v1:172.21.15.7:6789/0 conn(0x7f735c096e20 legacy=0x7f735c097220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:49.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.890+0000 7f735b7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:54:49.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.890+0000 7f735b7fe700 1 monclient: found mon.noname-c 2023-09-30T01:54:49.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.890+0000 7f735b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:49.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.890+0000 7f735b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3300/0 2023-09-30T01:54:49.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.890+0000 7f735b7fe700 1 -- 172.21.15.7:0/3154044661 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735c099720 con 0x7f735c096a50 2023-09-30T01:54:49.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.892+0000 7f735b7fe700 1 -- 172.21.15.7:0/3154044661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f735c096a50 msgr2=0x7f735c132680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:49.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.892+0000 7f735b7fe700 1 --2- 172.21.15.7:0/3154044661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f735c096a50 0x7f735c132680 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f735c097d30 tx=0x7f734c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2ea7de15635de76 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:49.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f735a7fc700 1 -- 172.21.15.7:0/3154044661 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c011920 con 0x7f735c096a50 2023-09-30T01:54:49.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f735a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:49.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f735a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:49.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f735a7fc700 10 monclient: dump: 2023-09-30T01:54:49.900 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:49.900 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:49.900 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.900 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.900 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:49.900 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:49.901 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:49.901 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:49.901 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:49.901 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:49.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f735a7fc700 10 monclient: _finish_auth 0 2023-09-30T01:54:49.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f735a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:49.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f7361c74700 5 monclient: authenticate success, global_id 4273 2023-09-30T01:54:49.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f735a7fc700 1 -- 172.21.15.7:0/3154044661 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734c011a80 con 0x7f735c096a50 2023-09-30T01:54:49.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f735a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:49.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f7361c74700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:49.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f7361c74700 4 set_mon_vals no callback set 2023-09-30T01:54:49.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f735a7fc700 1 -- 172.21.15.7:0/3154044661 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c011e20 con 0x7f735c096a50 2023-09-30T01:54:49.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f735a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:49.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f735a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:54:49.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f735a7fc700 10 monclient: dump: 2023-09-30T01:54:49.902 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:49.902 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:49.902 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.902 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.902 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:49.902 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:49.902 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:49.902 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:49.903 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:49.903 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:49.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f7361c74700 10 monclient: shutdown 2023-09-30T01:54:49.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f7361c74700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:49.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f7361c74700 1 -- 172.21.15.7:0/3154044661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f735c096a50 msgr2=0x7f735c132680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:49.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f7361c74700 1 --2- 172.21.15.7:0/3154044661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f735c096a50 0x7f735c132680 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f735c097d30 tx=0x7f734c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:49.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f7361c74700 1 -- 172.21.15.7:0/3154044661 shutdown_connections 2023-09-30T01:54:49.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f7361c74700 1 --2- 172.21.15.7:0/3154044661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f735c096a50 0x7f735c132680 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:49.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f7361c74700 1 -- 172.21.15.7:0/3154044661 >> 172.21.15.7:0/3154044661 conn(0x7f735c070cf0 msgr2=0x7f735c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:49.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f7361c74700 1 -- 172.21.15.7:0/3154044661 shutdown_connections 2023-09-30T01:54:49.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.893+0000 7f7361c74700 1 -- 172.21.15.7:0/3154044661 wait complete. 2023-09-30T01:54:49.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.894+0000 7f7361c74700 10 monclient: build_initial_monmap 2023-09-30T01:54:49.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.894+0000 7f7361c74700 10 monclient: monmap: 2023-09-30T01:54:49.904 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:49.904 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:49.904 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:49.904 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:49.904 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:49.904 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:49.904 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:49.904 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:49.904 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:49.904 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:49.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.894+0000 7f7361c74700 1 Processor -- start 2023-09-30T01:54:49.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.894+0000 7f7361c74700 1 -- start start 2023-09-30T01:54:49.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.894+0000 7f7361c74700 10 monclient: init 2023-09-30T01:54:49.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f7361c74700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:49.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f7361c74700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:54:49.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f7361c74700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f735c096a50 0x7f735c0a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:49.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f7361c74700 10 monclient(hunting): picked mon.noname-b con 0x7f735c096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:49.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f7361c74700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f735c096e20 0x7f735c09cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:49.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f7361c74700 10 monclient(hunting): picked mon.noname-a con 0x7f735c096e20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:49.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f7361c74700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f735c137710 0x7f735c09d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:49.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f7361c74700 10 monclient(hunting): picked mon.noname-c con 0x7f735c137710 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:49.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f7361c74700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:49.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f7361c74700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c09e010 con 0x7f735c096e20 2023-09-30T01:54:49.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f7361c74700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:49.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f7361c74700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c1334e0 con 0x7f735c096a50 2023-09-30T01:54:49.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f7361c74700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:49.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f7361c74700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735c131df0 con 0x7f735c137710 2023-09-30T01:54:49.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f7361c74700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:49.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f7361c74700 10 monclient(hunting): authenticate will time out at 1178.960428s 2023-09-30T01:54:49.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f735c096e20 0x7f735c09cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:49.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735bfff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f735c137710 0x7f735c09d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:49.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f735c096e20 0x7f735c09cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54856/0 (socket says 172.21.15.7:54856) 2023-09-30T01:54:49.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735affd700 1 -- 172.21.15.7:0/3983866012 learned_addr learned my addr 172.21.15.7:0/3983866012 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:49.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735bfff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f735c137710 0x7f735c09d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59492/0 (socket says 172.21.15.7:59492) 2023-09-30T01:54:49.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735bfff700 10 monclient(hunting): get_auth_request con 0x7f735c137710 auth_method 0 2023-09-30T01:54:49.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:49.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:49.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:49.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735affd700 10 monclient(hunting): get_auth_request con 0x7f735c096e20 auth_method 0 2023-09-30T01:54:49.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:49.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:49.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:49.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735b7fe700 1 --2- 172.21.15.7:0/3983866012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f735c096a50 0x7f735c0a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:49.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735b7fe700 10 monclient(hunting): get_auth_request con 0x7f735c096a50 auth_method 0 2023-09-30T01:54:49.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:49.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:49.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:49.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:49.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:49.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:49.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:49.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:49.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:49.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:49.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:49.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:49.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735affd700 10 monclient(hunting): handle_auth_done global_id 4272 payload 824 2023-09-30T01:54:49.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735affd700 1 -- 172.21.15.7:0/3983866012 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f735c137710 msgr2=0x7f735c09d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:49.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735affd700 1 --2- 172.21.15.7:0/3983866012 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f735c137710 0x7f735c09d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:49.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.895+0000 7f735affd700 1 -- 172.21.15.7:0/3983866012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f735c096a50 msgr2=0x7f735c0a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:49.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.896+0000 7f735affd700 1 --2- 172.21.15.7:0/3983866012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f735c096a50 0x7f735c0a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:49.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.896+0000 7f735affd700 10 monclient: _finish_hunting 0 2023-09-30T01:54:49.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.896+0000 7f735affd700 1 monclient: found mon.noname-a 2023-09-30T01:54:49.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.896+0000 7f735affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:49.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.896+0000 7f735affd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:49.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.896+0000 7f735affd700 1 -- 172.21.15.7:0/3983866012 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734c009420 con 0x7f735c096e20 2023-09-30T01:54:49.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.896+0000 7f735bfff700 1 --2- 172.21.15.7:0/3983866012 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f735c137710 0x7f735c09d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:49.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.896+0000 7f735b7fe700 1 --2- 172.21.15.7:0/3983866012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f735c096a50 0x7f735c0a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:49.910 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T01:54:49.910 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T01:54:49.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f735affd700 1 -- 172.21.15.7:0/3983866012 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f735c096e20 msgr2=0x7f735c09cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:49.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f735affd700 1 --2- 172.21.15.7:0/3983866012 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f735c096e20 0x7f735c09cb30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f735000c5e0 tx=0x7f7350004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:49.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f73437fe700 1 -- 172.21.15.7:0/3983866012 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7350005d40 con 0x7f735c096e20 2023-09-30T01:54:49.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f73437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:49.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f73437fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:49.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f73437fe700 10 monclient: dump: 2023-09-30T01:54:49.913 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:49.913 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:49.913 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:49.914 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f73437fe700 10 monclient: _finish_auth 0 2023-09-30T01:54:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f73437fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f7361c74700 5 monclient: authenticate success, global_id 4272 2023-09-30T01:54:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f7361c74700 10 monclient: _renew_subs 2023-09-30T01:54:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f73437fe700 1 -- 172.21.15.7:0/3983866012 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73500070d0 con 0x7f735c096e20 2023-09-30T01:54:49.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f7361c74700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:49.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f7361c74700 1 -- 172.21.15.7:0/3983866012 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f735c09ec70 con 0x7f735c096e20 2023-09-30T01:54:49.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f73437fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:49.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f73437fe700 1 -- 172.21.15.7:0/3983866012 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7350007490 con 0x7f735c096e20 2023-09-30T01:54:49.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f73437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:49.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f73437fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:49.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f73437fe700 10 monclient: dump: 2023-09-30T01:54:49.917 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:49.917 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:49.917 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.917 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.917 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:49.918 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:49.918 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:49.918 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:49.918 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:49.918 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:49.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f7359ffb700 4 set_mon_vals no callback set 2023-09-30T01:54:49.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f7361c74700 10 monclient: _renew_subs 2023-09-30T01:54:49.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f7361c74700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:49.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.898+0000 7f7361c74700 1 -- 172.21.15.7:0/3983866012 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f735c09e710 con 0x7f735c096e20 2023-09-30T01:54:49.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.899+0000 7f7361c74700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:49.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.899+0000 7f7361c74700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:49.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.899+0000 7f7361c74700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:49.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.899+0000 7f7361c74700 1 -- 172.21.15.7:0/3983866012 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f735c064c10 con 0x7f735c096e20 2023-09-30T01:54:49.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.900+0000 7f73437fe700 1 -- 172.21.15.7:0/3983866012 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f735001b3b0 con 0x7f735c096e20 2023-09-30T01:54:49.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.900+0000 7f73437fe700 1 --2- 172.21.15.7:0/3983866012 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f73480357e0 0x7f7348037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:49.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.900+0000 7f73437fe700 1 -- 172.21.15.7:0/3983866012 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f7350047580 con 0x7f735c096e20 2023-09-30T01:54:49.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.900+0000 7f735b7fe700 1 --2- 172.21.15.7:0/3983866012 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f73480357e0 0x7f7348037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:49.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.900+0000 7f735b7fe700 10 monclient: get_auth_request con 0x7f73480357e0 auth_method 0 2023-09-30T01:54:49.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.903+0000 7f735b7fe700 1 --2- 172.21.15.7:0/3983866012 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f73480357e0 0x7f7348037c90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f734c0093a0 tx=0x7f734c016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:49.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.908+0000 7f73437fe700 1 -- 172.21.15.7:0/3983866012 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f735001be60 con 0x7f735c096e20 2023-09-30T01:54:49.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.908+0000 7f73437fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:49.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.908+0000 7f73437fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:49.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.930+0000 7f519019e700 10 monclient: get_monmap_and_config 2023-09-30T01:54:49.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.930+0000 7f519019e700 10 monclient: build_initial_monmap 2023-09-30T01:54:49.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.930+0000 7f519019e700 10 monclient: monmap: 2023-09-30T01:54:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:49.931553+0000 2023-09-30T01:54:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:49.931553+0000 2023-09-30T01:54:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:49.931 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:49.932 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:49.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.931+0000 7f519019e700 1 Processor -- start 2023-09-30T01:54:49.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.932+0000 7f519019e700 1 -- start start 2023-09-30T01:54:49.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.932+0000 7f519019e700 10 monclient: init 2023-09-30T01:54:49.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.932+0000 7f519019e700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:49.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.932+0000 7f519019e700 10 monclient: _add_conns ranks=[1,4,2,0,3] 2023-09-30T01:54:49.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.932+0000 7f050d20c700 10 monclient: get_monmap_and_config 2023-09-30T01:54:49.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.932+0000 7f050d20c700 10 monclient: build_initial_monmap 2023-09-30T01:54:49.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.932+0000 7f050d20c700 10 monclient: monmap: 2023-09-30T01:54:49.933 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:49.934 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:49.934 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:49.933857+0000 2023-09-30T01:54:49.934 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:49.933857+0000 2023-09-30T01:54:49.934 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:49.934 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:49.934 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:49.934 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:49.934 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:49.934 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:49.934 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:49.934 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:49.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.932+0000 7f519019e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5188096a50 addr v1:172.21.15.50:6789/0 2023-09-30T01:54:49.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.932+0000 7f519019e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5188096e20 0x7f5188097220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:49.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.932+0000 7f519019e700 10 monclient(hunting): picked mon.noname-c con 0x7f5188096e20 addr v2:172.21.15.50:3300/0 2023-09-30T01:54:49.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.932+0000 7f519019e700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5188137710 0x7f5188139af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:49.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.932+0000 7f519019e700 10 monclient(hunting): picked mon.noname-b con 0x7f5188137710 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:49.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.932+0000 7f519019e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5188099470 con 0x7f5188096a50 2023-09-30T01:54:49.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.932+0000 7f519019e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:49.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.932+0000 7f519019e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5188132d90 con 0x7f5188096e20 2023-09-30T01:54:49.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.932+0000 7f519019e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:49.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.932+0000 7f519019e700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5188097b40 con 0x7f5188137710 2023-09-30T01:54:49.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.932+0000 7f519019e700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:49.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.932+0000 7f519019e700 10 monclient(hunting): authenticate will time out at 1178.997871s 2023-09-30T01:54:49.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518e73b700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5188137710 0x7f5188139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:49.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518e73b700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5188137710 0x7f5188139af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59498/0 (socket says 172.21.15.7:59498) 2023-09-30T01:54:49.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518e73b700 1 -- 172.21.15.7:0/1641763335 learned_addr learned my addr 172.21.15.7:0/1641763335 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:49.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518e73b700 10 monclient(hunting): get_auth_request con 0x7f5188137710 auth_method 0 2023-09-30T01:54:49.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518e73b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:49.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518e73b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:49.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518d739700 1 --2- 172.21.15.7:0/1641763335 >> v2:172.21.15.50:3300/0 conn(0x7f5188096e20 0x7f5188097220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:49.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518e73b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:49.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518d739700 10 monclient(hunting): get_auth_request con 0x7f5188096e20 auth_method 0 2023-09-30T01:54:49.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518d739700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:49.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518d739700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:49.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518d739700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:49.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518d739700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:49.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518d739700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:49.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518d739700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:49.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518e73b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:49.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518e73b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:49.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518e73b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:49.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518d739700 10 monclient(hunting): handle_auth_done global_id 4279 payload 274 2023-09-30T01:54:49.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518d739700 1 -- 172.21.15.7:0/1641763335 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5188137710 msgr2=0x7f5188139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:49.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518d739700 1 --2- 172.21.15.7:0/1641763335 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5188137710 0x7f5188139af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518d739700 1 -- 172.21.15.7:0/1641763335 >> v1:172.21.15.50:6789/0 conn(0x7f5188096a50 legacy=0x7f5188132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518d739700 10 monclient: _finish_hunting 0 2023-09-30T01:54:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518d739700 1 monclient: found mon.noname-c 2023-09-30T01:54:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518d739700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518d739700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3300/0 2023-09-30T01:54:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518d739700 1 -- 172.21.15.7:0/1641763335 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5188099160 con 0x7f5188096e20 2023-09-30T01:54:49.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f518e73b700 1 --2- 172.21.15.7:0/1641763335 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5188137710 0x7f5188139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:49.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f050d20c700 1 Processor -- start 2023-09-30T01:54:49.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f050d20c700 1 -- start start 2023-09-30T01:54:49.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.933+0000 7f050d20c700 10 monclient: init 2023-09-30T01:54:49.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050d20c700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:49.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050d20c700 10 monclient: _add_conns ranks=[1,0,3,2,4] 2023-09-30T01:54:49.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050d20c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0508096a50 addr v1:172.21.15.50:6789/0 2023-09-30T01:54:49.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050d20c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0508096e20 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:49.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050d20c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0508138720 0x7f050813ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:49.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050d20c700 10 monclient(hunting): picked mon.noname-a con 0x7f0508138720 addr v2:172.21.15.7:3300/0 2023-09-30T01:54:49.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050d20c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0508099440 con 0x7f0508096e20 2023-09-30T01:54:49.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050d20c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0508132d90 con 0x7f0508096a50 2023-09-30T01:54:49.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050d20c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:49.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050d20c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0508097b40 con 0x7f0508138720 2023-09-30T01:54:49.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050d20c700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:49.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050d20c700 10 monclient(hunting): authenticate will time out at 1178.999602s 2023-09-30T01:54:49.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050759e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0508138720 0x7f050813ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:49.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050659c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0508096e20 0x7f0508097220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58136/0 (socket says 172.21.15.7:58136) 2023-09-30T01:54:49.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050659c700 1 -- 172.21.15.7:0/4218191681 learned_addr learned my addr 172.21.15.7:0/4218191681 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:54:49.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050759e700 10 monclient(hunting): get_auth_request con 0x7f0508138720 auth_method 0 2023-09-30T01:54:49.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:49.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050759e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:49.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050759e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:49.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050759e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:49.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050759e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:49.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.934+0000 7f050759e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:49.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.935+0000 7f050759e700 10 monclient(hunting): handle_auth_done global_id 4275 payload 274 2023-09-30T01:54:49.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.935+0000 7f050759e700 1 -- 172.21.15.7:0/4218191681 >> v1:172.21.15.50:6789/0 conn(0x7f0508096a50 legacy=0x7f0508132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:49.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.935+0000 7f050759e700 1 -- 172.21.15.7:0/4218191681 >> v1:172.21.15.7:6789/0 conn(0x7f0508096e20 legacy=0x7f0508097220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:49.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.935+0000 7f050759e700 10 monclient: _finish_hunting 0 2023-09-30T01:54:49.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.935+0000 7f050759e700 1 monclient: found mon.noname-a 2023-09-30T01:54:49.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.935+0000 7f050759e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:49.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.935+0000 7f050759e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:49.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.935+0000 7f050759e700 1 -- 172.21.15.7:0/4218191681 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0508099160 con 0x7f0508138720 2023-09-30T01:54:49.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.935+0000 7f518d739700 1 -- 172.21.15.7:0/1641763335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5188096e20 msgr2=0x7f5188097220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:49.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.935+0000 7f518d739700 1 --2- 172.21.15.7:0/1641763335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5188096e20 0x7f5188097220 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f51780098e0 tx=0x7f51780091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ca0c1576968fefe server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:49.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.936+0000 7f518cf38700 1 -- 172.21.15.7:0/1641763335 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5178011840 con 0x7f5188096e20 2023-09-30T01:54:49.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.936+0000 7f518cf38700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:49.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.936+0000 7f518cf38700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:49.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.936+0000 7f518cf38700 10 monclient: dump: 2023-09-30T01:54:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:49.947 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.936+0000 7f518cf38700 10 monclient: _finish_auth 0 2023-09-30T01:54:49.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.936+0000 7f518cf38700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.936+0000 7f518cf38700 1 -- 172.21.15.7:0/1641763335 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51780119a0 con 0x7f5188096e20 2023-09-30T01:54:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.936+0000 7f519019e700 5 monclient: authenticate success, global_id 4279 2023-09-30T01:54:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.936+0000 7f519019e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.936+0000 7f518cf38700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.936+0000 7f518cf38700 1 -- 172.21.15.7:0/1641763335 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5178011d40 con 0x7f5188096e20 2023-09-30T01:54:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.936+0000 7f519019e700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.936+0000 7f519019e700 4 set_mon_vals no callback set 2023-09-30T01:54:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.936+0000 7f518cf38700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.936+0000 7f518cf38700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:54:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.936+0000 7f518cf38700 10 monclient: dump: 2023-09-30T01:54:49.948 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:49.949 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.936+0000 7f519019e700 10 monclient: shutdown 2023-09-30T01:54:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f519019e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f519019e700 1 -- 172.21.15.7:0/1641763335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5188096e20 msgr2=0x7f5188097220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:49.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f519019e700 1 --2- 172.21.15.7:0/1641763335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5188096e20 0x7f5188097220 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f51780098e0 tx=0x7f51780091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:49.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f519019e700 1 -- 172.21.15.7:0/1641763335 shutdown_connections 2023-09-30T01:54:49.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f519019e700 1 --2- 172.21.15.7:0/1641763335 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5188137710 0x7f5188139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:49.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f519019e700 1 --2- 172.21.15.7:0/1641763335 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5188096e20 0x7f5188097220 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:49.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f519019e700 1 -- 172.21.15.7:0/1641763335 >> 172.21.15.7:0/1641763335 conn(0x7f5188070cf0 msgr2=0x7f51880710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:49.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f519019e700 1 -- 172.21.15.7:0/1641763335 shutdown_connections 2023-09-30T01:54:49.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f519019e700 1 -- 172.21.15.7:0/1641763335 wait complete. 2023-09-30T01:54:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f050759e700 1 -- 172.21.15.7:0/4218191681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0508138720 msgr2=0x7f050813ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f050759e700 1 --2- 172.21.15.7:0/4218191681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0508138720 0x7f050813ab00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f04f8009e60 tx=0x7f04f8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62cf21148dcebb06 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f0505d9b700 1 -- 172.21.15.7:0/4218191681 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f800e070 con 0x7f0508138720 2023-09-30T01:54:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f0505d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f0505d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:49.951 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f0505d9b700 10 monclient: dump: 2023-09-30T01:54:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:49.952 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f0505d9b700 10 monclient: _finish_auth 0 2023-09-30T01:54:49.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f0505d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f0505d9b700 1 -- 172.21.15.7:0/4218191681 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f8012430 con 0x7f0508138720 2023-09-30T01:54:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f050d20c700 5 monclient: authenticate success, global_id 4275 2023-09-30T01:54:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f050d20c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f0505d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f050d20c700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f050d20c700 4 set_mon_vals no callback set 2023-09-30T01:54:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f0505d9b700 1 -- 172.21.15.7:0/4218191681 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f800bea0 con 0x7f0508138720 2023-09-30T01:54:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f0505d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f0505d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:49.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f0505d9b700 10 monclient: dump: 2023-09-30T01:54:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:49.954 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:49.954 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f519019e700 10 monclient: build_initial_monmap 2023-09-30T01:54:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.937+0000 7f519019e700 10 monclient: monmap: 2023-09-30T01:54:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:49.956 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:49.956 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:49.956 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:49.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.938+0000 7f050d20c700 10 monclient: shutdown 2023-09-30T01:54:49.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.938+0000 7f050d20c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:49.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.938+0000 7f050d20c700 1 -- 172.21.15.7:0/4218191681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0508138720 msgr2=0x7f050813ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:49.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.938+0000 7f050d20c700 1 --2- 172.21.15.7:0/4218191681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0508138720 0x7f050813ab00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f04f8009e60 tx=0x7f04f8009780 comp rx=0 tx=0).stop 2023-09-30T01:54:49.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.938+0000 7f050d20c700 1 -- 172.21.15.7:0/4218191681 shutdown_connections 2023-09-30T01:54:49.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.938+0000 7f050d20c700 1 --2- 172.21.15.7:0/4218191681 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0508138720 0x7f050813ab00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:49.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.938+0000 7f050d20c700 1 -- 172.21.15.7:0/4218191681 >> 172.21.15.7:0/4218191681 conn(0x7f0508070cf0 msgr2=0x7f05080710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:49.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.938+0000 7f519019e700 1 Processor -- start 2023-09-30T01:54:49.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.938+0000 7f050d20c700 1 -- 172.21.15.7:0/4218191681 shutdown_connections 2023-09-30T01:54:49.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.938+0000 7f519019e700 1 -- start start 2023-09-30T01:54:49.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.938+0000 7f519019e700 10 monclient: init 2023-09-30T01:54:49.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f519019e700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:49.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f519019e700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:54:49.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f519019e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5188096a50 0x7f518812fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:49.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f519019e700 10 monclient(hunting): picked mon.noname-b con 0x7f5188096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:49.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f519019e700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5188096e20 0x7f5188130170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:49.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f519019e700 10 monclient(hunting): picked mon.noname-c con 0x7f5188096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:49.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f519019e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5188137710 0x7f518809e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:49.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f519019e700 10 monclient(hunting): picked mon.noname-a con 0x7f5188137710 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:49.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f519019e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:49.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f519019e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51881334e0 con 0x7f5188137710 2023-09-30T01:54:49.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f519019e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:49.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f519019e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5188131e60 con 0x7f5188096a50 2023-09-30T01:54:49.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f519019e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:49.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f519019e700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f518811c0b0 con 0x7f5188096e20 2023-09-30T01:54:49.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f519019e700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:49.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f519019e700 10 monclient(hunting): authenticate will time out at 1179.004330s 2023-09-30T01:54:49.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518d739700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5188096e20 0x7f5188130170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:49.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518e73b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5188137710 0x7f518809e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:49.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518d739700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5188096e20 0x7f5188130170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59500/0 (socket says 172.21.15.7:59500) 2023-09-30T01:54:49.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518d739700 1 -- 172.21.15.7:0/1877411813 learned_addr learned my addr 172.21.15.7:0/1877411813 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:49.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518d739700 10 monclient(hunting): get_auth_request con 0x7f5188096e20 auth_method 0 2023-09-30T01:54:49.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518d739700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:49.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518d739700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:49.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518d739700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:49.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518e73b700 10 monclient(hunting): get_auth_request con 0x7f5188137710 auth_method 0 2023-09-30T01:54:49.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518e73b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:49.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518e73b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:49.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518e73b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:49.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518e73b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:49.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518e73b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:49.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518e73b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:49.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518df3a700 1 --2- 172.21.15.7:0/1877411813 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5188096a50 0x7f518812fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:49.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518df3a700 10 monclient(hunting): get_auth_request con 0x7f5188096a50 auth_method 0 2023-09-30T01:54:49.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518df3a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:49.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518df3a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:49.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518df3a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:49.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518df3a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:49.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518df3a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:49.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518df3a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:49.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518e73b700 10 monclient(hunting): handle_auth_done global_id 4278 payload 824 2023-09-30T01:54:49.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518e73b700 1 -- 172.21.15.7:0/1877411813 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5188096e20 msgr2=0x7f5188130170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:49.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518e73b700 1 --2- 172.21.15.7:0/1877411813 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5188096e20 0x7f5188130170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:49.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518e73b700 1 -- 172.21.15.7:0/1877411813 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5188096a50 msgr2=0x7f518812fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:49.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518e73b700 1 --2- 172.21.15.7:0/1877411813 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5188096a50 0x7f518812fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:49.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518e73b700 10 monclient: _finish_hunting 0 2023-09-30T01:54:49.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518e73b700 1 monclient: found mon.noname-a 2023-09-30T01:54:49.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518e73b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:49.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518e73b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:49.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f518e73b700 1 -- 172.21.15.7:0/1877411813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5180009ae0 con 0x7f5188137710 2023-09-30T01:54:49.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.939+0000 7f050d20c700 1 -- 172.21.15.7:0/4218191681 wait complete. 2023-09-30T01:54:49.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.940+0000 7f050d20c700 10 monclient: build_initial_monmap 2023-09-30T01:54:49.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.940+0000 7f050d20c700 10 monclient: monmap: 2023-09-30T01:54:49.966 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:49.967 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:49.967 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:49.967 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:49.967 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:49.967 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:49.967 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:49.967 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:49.967 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:49.967 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:49.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.940+0000 7f050d20c700 1 Processor -- start 2023-09-30T01:54:49.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.940+0000 7f050d20c700 1 -- start start 2023-09-30T01:54:49.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.940+0000 7f050d20c700 10 monclient: init 2023-09-30T01:54:49.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050d20c700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:49.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050d20c700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:54:49.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050d20c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0508096a50 0x7f050812c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:49.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050d20c700 10 monclient(hunting): picked mon.noname-c con 0x7f0508096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:49.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050d20c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0508096e20 0x7f050806dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:49.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050d20c700 10 monclient(hunting): picked mon.noname-a con 0x7f0508096e20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:49.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050d20c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0508138720 0x7f050806e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:49.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050d20c700 10 monclient(hunting): picked mon.noname-b con 0x7f0508138720 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:49.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050d20c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:49.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050d20c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0508135cc0 con 0x7f0508096e20 2023-09-30T01:54:49.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050d20c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050d20c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05080a6550 con 0x7f0508138720 2023-09-30T01:54:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050d20c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050d20c700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0508135000 con 0x7f0508096a50 2023-09-30T01:54:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050d20c700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050d20c700 10 monclient(hunting): authenticate will time out at 1179.006220s 2023-09-30T01:54:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0508096e20 0x7f050806dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f0506d9d700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0508096a50 0x7f050812c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050659c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0508096e20 0x7f050806dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54866/0 (socket says 172.21.15.7:54866) 2023-09-30T01:54:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f0506d9d700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0508096a50 0x7f050812c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59514/0 (socket says 172.21.15.7:59514) 2023-09-30T01:54:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050659c700 1 -- 172.21.15.7:0/386531023 learned_addr learned my addr 172.21.15.7:0/386531023 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050659c700 10 monclient(hunting): get_auth_request con 0x7f0508096e20 auth_method 0 2023-09-30T01:54:49.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:49.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:49.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:49.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f0506d9d700 10 monclient(hunting): get_auth_request con 0x7f0508096a50 auth_method 0 2023-09-30T01:54:49.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f0506d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:49.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f0506d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:49.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f0506d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:49.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f0506d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:49.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f0506d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:49.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050759e700 1 --2- 172.21.15.7:0/386531023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0508138720 0x7f050806e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:49.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f0506d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:49.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:49.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:49.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:49.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050759e700 10 monclient(hunting): get_auth_request con 0x7f0508138720 auth_method 0 2023-09-30T01:54:49.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:49.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050759e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:49.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050759e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:49.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050759e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:49.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050759e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:49.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050759e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:49.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050659c700 10 monclient(hunting): handle_auth_done global_id 4281 payload 824 2023-09-30T01:54:49.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050659c700 1 -- 172.21.15.7:0/386531023 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0508096a50 msgr2=0x7f050812c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:49.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050659c700 1 --2- 172.21.15.7:0/386531023 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0508096a50 0x7f050812c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:49.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050659c700 1 -- 172.21.15.7:0/386531023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0508138720 msgr2=0x7f050806e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:49.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050659c700 1 --2- 172.21.15.7:0/386531023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0508138720 0x7f050806e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:49.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050659c700 10 monclient: _finish_hunting 0 2023-09-30T01:54:49.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050659c700 1 monclient: found mon.noname-a 2023-09-30T01:54:49.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:49.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050659c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:49.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.941+0000 7f050659c700 1 -- 172.21.15.7:0/386531023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f8009a20 con 0x7f0508096e20 2023-09-30T01:54:49.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f518e73b700 1 -- 172.21.15.7:0/1877411813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5188137710 msgr2=0x7f518809e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:49.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f518e73b700 1 --2- 172.21.15.7:0/1877411813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5188137710 0x7f518809e800 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5180009f10 tx=0x7f518000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:49.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f517dffb700 1 -- 172.21.15.7:0/1877411813 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5180018070 con 0x7f5188137710 2023-09-30T01:54:49.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f517dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:49.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f517dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:49.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f517dffb700 10 monclient: dump: 2023-09-30T01:54:49.974 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:49.974 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:49.974 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.974 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.974 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:49.974 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:49.974 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:49.975 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:49.975 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:49.975 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:49.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f517dffb700 10 monclient: _finish_auth 0 2023-09-30T01:54:49.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f517dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:49.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f517dffb700 1 -- 172.21.15.7:0/1877411813 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f518000f040 con 0x7f5188137710 2023-09-30T01:54:49.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f519019e700 5 monclient: authenticate success, global_id 4278 2023-09-30T01:54:49.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f519019e700 10 monclient: _renew_subs 2023-09-30T01:54:49.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f519019e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:49.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f519019e700 1 -- 172.21.15.7:0/1877411813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5178009420 con 0x7f5188137710 2023-09-30T01:54:49.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f517dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:49.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f517dffb700 1 -- 172.21.15.7:0/1877411813 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51800092e0 con 0x7f5188137710 2023-09-30T01:54:49.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f517dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:49.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f517dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:49.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f517dffb700 10 monclient: dump: 2023-09-30T01:54:49.976 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:49.976 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:49.976 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.976 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.976 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:49.976 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:49.976 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:49.977 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:49.977 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:49.977 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:49.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f519019e700 10 monclient: _renew_subs 2023-09-30T01:54:49.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f519019e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:49.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f519019e700 1 -- 172.21.15.7:0/1877411813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51880a0be0 con 0x7f5188137710 2023-09-30T01:54:49.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.942+0000 7f517ffff700 4 set_mon_vals no callback set 2023-09-30T01:54:49.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.943+0000 7f517dffb700 1 -- 172.21.15.7:0/1877411813 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f5180003020 con 0x7f5188137710 2023-09-30T01:54:49.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f519019e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:49.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f519019e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:49.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f519019e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:49.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f517dffb700 1 --2- 172.21.15.7:0/1877411813 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f516c035830 0x7f516c037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:49.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f050659c700 1 -- 172.21.15.7:0/386531023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0508096e20 msgr2=0x7f050806dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:49.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f050659c700 1 --2- 172.21.15.7:0/386531023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0508096e20 0x7f050806dcf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00c5e0 tx=0x7f04fc004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:49.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f519019e700 1 -- 172.21.15.7:0/1877411813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f518811c240 con 0x7f5188137710 2023-09-30T01:54:49.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f517dffb700 1 -- 172.21.15.7:0/1877411813 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f5180047b50 con 0x7f5188137710 2023-09-30T01:54:49.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f04eeffd700 1 -- 172.21.15.7:0/386531023 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc005d40 con 0x7f0508096e20 2023-09-30T01:54:49.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f04eeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:49.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f04eeffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:49.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f04eeffd700 10 monclient: dump: 2023-09-30T01:54:49.980 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:49.980 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:49.980 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.980 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.980 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:49.980 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:49.980 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:49.980 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:49.980 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:49.980 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:49.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f04eeffd700 10 monclient: _finish_auth 0 2023-09-30T01:54:49.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f04eeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:49.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f04eeffd700 1 -- 172.21.15.7:0/386531023 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04fc0070d0 con 0x7f0508096e20 2023-09-30T01:54:49.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f050d20c700 5 monclient: authenticate success, global_id 4281 2023-09-30T01:54:49.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f050d20c700 10 monclient: _renew_subs 2023-09-30T01:54:49.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f050d20c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:49.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f050d20c700 1 -- 172.21.15.7:0/386531023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0508135620 con 0x7f0508096e20 2023-09-30T01:54:49.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f050d20c700 10 monclient: _renew_subs 2023-09-30T01:54:49.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f050d20c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:49.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f050d20c700 1 -- 172.21.15.7:0/386531023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05081334e0 con 0x7f0508096e20 2023-09-30T01:54:49.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f04eeffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:49.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f04eeffd700 1 -- 172.21.15.7:0/386531023 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc007490 con 0x7f0508096e20 2023-09-30T01:54:49.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f04eeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:49.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f04eeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:49.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f04eeffd700 10 monclient: dump: 2023-09-30T01:54:49.982 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:49.982 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:49.982 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.982 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:49.982 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:49.983 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:49.983 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:49.983 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:49.983 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:49.983 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:49.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.944+0000 7f050559a700 4 set_mon_vals no callback set 2023-09-30T01:54:49.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.945+0000 7f518df3a700 1 --2- 172.21.15.7:0/1877411813 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f516c035830 0x7f516c037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:49.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.945+0000 7f518df3a700 10 monclient: get_auth_request con 0x7f516c035830 auth_method 0 2023-09-30T01:54:49.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.945+0000 7f050d20c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:49.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.945+0000 7f050d20c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:49.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.945+0000 7f050d20c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:49.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.945+0000 7f050d20c700 1 -- 172.21.15.7:0/386531023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04e80050a0 con 0x7f0508096e20 2023-09-30T01:54:49.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.947+0000 7f518df3a700 1 --2- 172.21.15.7:0/1877411813 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f516c035830 0x7f516c037ce0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5184003a30 tx=0x7f5184007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:49.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.951+0000 7f04eeffd700 1 -- 172.21.15.7:0/386531023 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f04fc01b3b0 con 0x7f0508096e20 2023-09-30T01:54:49.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.951+0000 7f04eeffd700 1 --2- 172.21.15.7:0/386531023 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f04f4035830 0x7f04f4037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:49.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.951+0000 7f04eeffd700 1 -- 172.21.15.7:0/386531023 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f04fc0475a0 con 0x7f0508096e20 2023-09-30T01:54:49.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.951+0000 7f517dffb700 1 -- 172.21.15.7:0/1877411813 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5180012750 con 0x7f5188137710 2023-09-30T01:54:49.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.951+0000 7f517dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:49.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.951+0000 7f517dffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:49.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.952+0000 7f0506d9d700 1 --2- 172.21.15.7:0/386531023 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f04f4035830 0x7f04f4037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:49.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.952+0000 7f0506d9d700 10 monclient: get_auth_request con 0x7f04f4035830 auth_method 0 2023-09-30T01:54:49.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.954+0000 7f0506d9d700 1 --2- 172.21.15.7:0/386531023 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f04f4035830 0x7f04f4037ce0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0500009af0 tx=0x7f0500009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:49.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.957+0000 7f04eeffd700 1 -- 172.21.15.7:0/386531023 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f04fc0194f0 con 0x7f0508096e20 2023-09-30T01:54:49.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.957+0000 7f04eeffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:49.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:49.957+0000 7f04eeffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:50.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.104+0000 7f21f55e0700 10 monclient: get_monmap_and_config 2023-09-30T01:54:50.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.104+0000 7f21f55e0700 10 monclient: build_initial_monmap 2023-09-30T01:54:50.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.104+0000 7f21f55e0700 10 monclient: monmap: 2023-09-30T01:54:50.105 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:50.105 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:50.105 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:50.105710+0000 2023-09-30T01:54:50.105 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:50.105710+0000 2023-09-30T01:54:50.105 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:50.106 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.105+0000 7f21f55e0700 1 Processor -- start 2023-09-30T01:54:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.105+0000 7f21f55e0700 1 -- start start 2023-09-30T01:54:50.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.105+0000 7f21f55e0700 10 monclient: init 2023-09-30T01:54:50.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.105+0000 7f21f55e0700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:50.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.105+0000 7f21f55e0700 10 monclient: _add_conns ranks=[0,4,2,1,3] 2023-09-30T01:54:50.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.105+0000 7ff8edfdb700 10 monclient: get_monmap_and_config 2023-09-30T01:54:50.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.105+0000 7ff8edfdb700 10 monclient: build_initial_monmap 2023-09-30T01:54:50.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.106+0000 7ff8edfdb700 10 monclient: monmap: 2023-09-30T01:54:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:50.107037+0000 2023-09-30T01:54:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:50.107037+0000 2023-09-30T01:54:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:50.108 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.105+0000 7f21f55e0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f21f0096a50 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:50.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.105+0000 7f21f55e0700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f21f0096e20 0x7f21f0097220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.105+0000 7f21f55e0700 10 monclient(hunting): picked mon.noname-c con 0x7f21f0096e20 addr v2:172.21.15.50:3300/0 2023-09-30T01:54:50.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.105+0000 7f21f55e0700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f21f0137710 0x7f21f0139af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.105+0000 7f21f55e0700 10 monclient(hunting): picked mon.noname-b con 0x7f21f0137710 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:50.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.105+0000 7f21f55e0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21f0099470 con 0x7f21f0096a50 2023-09-30T01:54:50.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.105+0000 7f21f55e0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:50.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.105+0000 7f21f55e0700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21f0132d90 con 0x7f21f0096e20 2023-09-30T01:54:50.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.105+0000 7f21f55e0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:50.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.105+0000 7f21f55e0700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f0097b40 con 0x7f21f0137710 2023-09-30T01:54:50.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.105+0000 7f21f55e0700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:50.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.105+0000 7f21f55e0700 10 monclient(hunting): authenticate will time out at 1179.170967s 2023-09-30T01:54:50.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.106+0000 7f21eeffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f21f0096a50 0x7f21f0132680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58146/0 (socket says 172.21.15.7:58146) 2023-09-30T01:54:50.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.106+0000 7f21eeffd700 1 -- 172.21.15.7:0/2395635461 learned_addr learned my addr 172.21.15.7:0/2395635461 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:54:50.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.106+0000 7f21ee7fc700 1 --2- 172.21.15.7:0/2395635461 >> v2:172.21.15.50:3300/0 conn(0x7f21f0096e20 0x7f21f0097220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.106+0000 7f21ee7fc700 10 monclient(hunting): get_auth_request con 0x7f21f0096e20 auth_method 0 2023-09-30T01:54:50.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.106+0000 7f21ee7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:50.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.106+0000 7f21ee7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:50.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.106+0000 7f21ee7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:50.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.106+0000 7f21ee7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:50.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.106+0000 7f21ee7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:50.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.106+0000 7f21ee7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:50.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.106+0000 7f21ee7fc700 10 monclient(hunting): handle_auth_done global_id 4288 payload 274 2023-09-30T01:54:50.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.106+0000 7f21ee7fc700 1 -- 172.21.15.7:0/2395635461 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f21f0137710 msgr2=0x7f21f0139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:50.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.106+0000 7f21ee7fc700 1 --2- 172.21.15.7:0/2395635461 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f21f0137710 0x7f21f0139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.106+0000 7f21ee7fc700 1 -- 172.21.15.7:0/2395635461 >> v1:172.21.15.7:6789/0 conn(0x7f21f0096a50 legacy=0x7f21f0132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:50.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.106+0000 7f21ee7fc700 10 monclient: _finish_hunting 0 2023-09-30T01:54:50.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.106+0000 7f21ee7fc700 1 monclient: found mon.noname-c 2023-09-30T01:54:50.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.106+0000 7f21ee7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:50.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.106+0000 7f21ee7fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3300/0 2023-09-30T01:54:50.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.106+0000 7f21ee7fc700 1 -- 172.21.15.7:0/2395635461 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f0099160 con 0x7f21f0096e20 2023-09-30T01:54:50.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.106+0000 7ff8edfdb700 1 Processor -- start 2023-09-30T01:54:50.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.107+0000 7ff8edfdb700 1 -- start start 2023-09-30T01:54:50.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.107+0000 7ff8edfdb700 10 monclient: init 2023-09-30T01:54:50.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.107+0000 7ff8edfdb700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:50.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.107+0000 7ff8edfdb700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2023-09-30T01:54:50.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.107+0000 7ff8edfdb700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff8e8098540 0x7ff8e81329e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.107+0000 7ff8edfdb700 10 monclient(hunting): picked mon.noname-c con 0x7ff8e8098540 addr v2:172.21.15.50:3300/0 2023-09-30T01:54:50.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.107+0000 7ff8edfdb700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8e8098910 0x7ff8e8098ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.107+0000 7ff8edfdb700 10 monclient(hunting): picked mon.noname-b con 0x7ff8e8098910 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:50.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.107+0000 7ff8edfdb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff8e812ebc0 0x7ff8e812f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.107+0000 7ff8edfdb700 10 monclient(hunting): picked mon.noname-a con 0x7ff8e812ebc0 addr v2:172.21.15.7:3300/0 2023-09-30T01:54:50.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.107+0000 7ff8edfdb700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:50.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.107+0000 7ff8edfdb700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e80a0f40 con 0x7ff8e812ebc0 2023-09-30T01:54:50.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.107+0000 7ff8edfdb700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:50.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.107+0000 7ff8edfdb700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e8069f10 con 0x7ff8e8098540 2023-09-30T01:54:50.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.107+0000 7ff8edfdb700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:50.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.107+0000 7ff8edfdb700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e80a05c0 con 0x7ff8e8098910 2023-09-30T01:54:50.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.107+0000 7ff8edfdb700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:50.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.107+0000 7ff8edfdb700 10 monclient(hunting): authenticate will time out at 1179.172915s 2023-09-30T01:54:50.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8e7fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8e8098910 0x7ff8e8098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8e7fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8e8098910 0x7ff8e8098ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59536/0 (socket says 172.21.15.7:59536) 2023-09-30T01:54:50.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8e7fff700 1 -- 172.21.15.7:0/812081822 learned_addr learned my addr 172.21.15.7:0/812081822 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:50.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8e7fff700 10 monclient(hunting): get_auth_request con 0x7ff8e8098910 auth_method 0 2023-09-30T01:54:50.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8e7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:50.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8e7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:50.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8e7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:50.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8ecfd9700 1 --2- 172.21.15.7:0/812081822 >> v2:172.21.15.50:3300/0 conn(0x7ff8e8098540 0x7ff8e81329e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8ecfd9700 10 monclient(hunting): get_auth_request con 0x7ff8e8098540 auth_method 0 2023-09-30T01:54:50.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8ecfd9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:50.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8ecfd9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:50.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8ecfd9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:50.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8e7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:50.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8e7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:50.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8e7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:50.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8ecfd9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:50.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8ecfd9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:50.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8ecfd9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:50.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8ed7da700 1 --2- 172.21.15.7:0/812081822 >> v2:172.21.15.7:3300/0 conn(0x7ff8e812ebc0 0x7ff8e812f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8ed7da700 10 monclient(hunting): get_auth_request con 0x7ff8e812ebc0 auth_method 0 2023-09-30T01:54:50.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8ed7da700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:50.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8ed7da700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:50.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8ed7da700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:50.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8ed7da700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:50.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8ed7da700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:50.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8ed7da700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:50.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8e7fff700 10 monclient(hunting): handle_auth_done global_id 4289 payload 274 2023-09-30T01:54:50.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8e7fff700 1 -- 172.21.15.7:0/812081822 >> v2:172.21.15.50:3300/0 conn(0x7ff8e8098540 msgr2=0x7ff8e81329e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:50.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8e7fff700 1 --2- 172.21.15.7:0/812081822 >> v2:172.21.15.50:3300/0 conn(0x7ff8e8098540 0x7ff8e81329e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8e7fff700 1 -- 172.21.15.7:0/812081822 >> v2:172.21.15.7:3300/0 conn(0x7ff8e812ebc0 msgr2=0x7ff8e812f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:50.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8e7fff700 1 --2- 172.21.15.7:0/812081822 >> v2:172.21.15.7:3300/0 conn(0x7ff8e812ebc0 0x7ff8e812f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8e7fff700 10 monclient: _finish_hunting 0 2023-09-30T01:54:50.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8e7fff700 1 monclient: found mon.noname-b 2023-09-30T01:54:50.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8e7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:50.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8e7fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:54:50.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8e7fff700 1 -- 172.21.15.7:0/812081822 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e8096a50 con 0x7ff8e8098910 2023-09-30T01:54:50.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7ff8ecfd9700 1 --2- 172.21.15.7:0/812081822 >> v2:172.21.15.50:3300/0 conn(0x7ff8e8098540 0x7ff8e81329e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:50.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.108+0000 7f21ee7fc700 1 -- 172.21.15.7:0/2395635461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21f0096e20 msgr2=0x7f21f0097220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:50.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.109+0000 7f21ee7fc700 1 --2- 172.21.15.7:0/2395635461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21f0096e20 0x7f21f0097220 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f21e40098e0 tx=0x7f21e40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfb969968ca440a0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:50.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.109+0000 7f21edffb700 1 -- 172.21.15.7:0/2395635461 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e400ba30 con 0x7f21f0096e20 2023-09-30T01:54:50.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.109+0000 7f21edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:50.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.109+0000 7f21edffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:50.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.109+0000 7f21edffb700 10 monclient: dump: 2023-09-30T01:54:50.122 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:50.122 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:50.122 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.122 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.122 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:50.122 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.122 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:50.122 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:50.122 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:50.122 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.109+0000 7f21edffb700 10 monclient: _finish_auth 0 2023-09-30T01:54:50.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.109+0000 7f21edffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:50.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.109+0000 7f21f55e0700 5 monclient: authenticate success, global_id 4288 2023-09-30T01:54:50.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.109+0000 7f21edffb700 1 -- 172.21.15.7:0/2395635461 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e400bb90 con 0x7f21f0096e20 2023-09-30T01:54:50.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.109+0000 7f21edffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:50.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.109+0000 7f21edffb700 1 -- 172.21.15.7:0/2395635461 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4011420 con 0x7f21f0096e20 2023-09-30T01:54:50.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.109+0000 7f21f55e0700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:50.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.109+0000 7f21f55e0700 4 set_mon_vals no callback set 2023-09-30T01:54:50.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.109+0000 7f21edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:50.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.109+0000 7f21edffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:54:50.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.109+0000 7f21edffb700 10 monclient: dump: 2023-09-30T01:54:50.123 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:50.123 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:50.124 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.124 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.124 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:50.124 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.124 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:50.124 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:50.124 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:50.124 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.110+0000 7f21f55e0700 10 monclient: shutdown 2023-09-30T01:54:50.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.110+0000 7f21f55e0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:50.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.110+0000 7f21f55e0700 1 -- 172.21.15.7:0/2395635461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21f0096e20 msgr2=0x7f21f0097220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.110+0000 7f21f55e0700 1 --2- 172.21.15.7:0/2395635461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21f0096e20 0x7f21f0097220 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f21e40098e0 tx=0x7f21e40091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.110+0000 7f21f55e0700 1 -- 172.21.15.7:0/2395635461 shutdown_connections 2023-09-30T01:54:50.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.110+0000 7f21f55e0700 1 --2- 172.21.15.7:0/2395635461 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f21f0137710 0x7f21f0139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.110+0000 7f21f55e0700 1 --2- 172.21.15.7:0/2395635461 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21f0096e20 0x7f21f0097220 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.110+0000 7f21f55e0700 1 -- 172.21.15.7:0/2395635461 >> 172.21.15.7:0/2395635461 conn(0x7f21f0070cf0 msgr2=0x7f21f00710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:50.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.110+0000 7f21f55e0700 1 -- 172.21.15.7:0/2395635461 shutdown_connections 2023-09-30T01:54:50.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.110+0000 7f21f55e0700 1 -- 172.21.15.7:0/2395635461 wait complete. 2023-09-30T01:54:50.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.110+0000 7ff8e7fff700 1 -- 172.21.15.7:0/812081822 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8e8098910 msgr2=0x7ff8e8098ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:50.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.110+0000 7ff8e7fff700 1 --2- 172.21.15.7:0/812081822 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8e8098910 0x7ff8e8098ce0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8071cb0 tx=0x7ff8d4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30e0bf6da2ed1396 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:50.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.110+0000 7ff8e77fe700 1 -- 172.21.15.7:0/812081822 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d400b850 con 0x7ff8e8098910 2023-09-30T01:54:50.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8e77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:50.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8e77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:54:50.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8e77fe700 10 monclient: dump: 2023-09-30T01:54:50.126 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:50.126 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:50.127 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.127 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.127 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:50.127 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.127 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:50.127 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:50.127 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:50.127 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8e77fe700 10 monclient: _finish_auth 0 2023-09-30T01:54:50.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8e77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:50.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8edfdb700 5 monclient: authenticate success, global_id 4289 2023-09-30T01:54:50.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8edfdb700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:50.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8e77fe700 1 -- 172.21.15.7:0/812081822 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d400b9b0 con 0x7ff8e8098910 2023-09-30T01:54:50.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8e77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:50.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8edfdb700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:50.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8edfdb700 4 set_mon_vals no callback set 2023-09-30T01:54:50.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8e77fe700 1 -- 172.21.15.7:0/812081822 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d400bd70 con 0x7ff8e8098910 2023-09-30T01:54:50.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8e77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:50.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8e77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:50.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8e77fe700 10 monclient: dump: 2023-09-30T01:54:50.128 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:50.128 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:50.128 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.128 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.128 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:50.129 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.129 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:50.129 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:50.129 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:50.129 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7f21f55e0700 10 monclient: build_initial_monmap 2023-09-30T01:54:50.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7f21f55e0700 10 monclient: monmap: 2023-09-30T01:54:50.129 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:50.130 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:50.130 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:50.130 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:50.130 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:50.130 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.130 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:50.130 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:50.130 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:50.130 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8edfdb700 10 monclient: shutdown 2023-09-30T01:54:50.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8edfdb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:50.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8edfdb700 1 -- 172.21.15.7:0/812081822 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8e8098910 msgr2=0x7ff8e8098ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8edfdb700 1 --2- 172.21.15.7:0/812081822 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8e8098910 0x7ff8e8098ce0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8071cb0 tx=0x7ff8d4009250 comp rx=0 tx=0).stop 2023-09-30T01:54:50.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7f21f55e0700 1 Processor -- start 2023-09-30T01:54:50.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8edfdb700 1 -- 172.21.15.7:0/812081822 shutdown_connections 2023-09-30T01:54:50.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8edfdb700 1 --2- 172.21.15.7:0/812081822 >> v2:172.21.15.7:3300/0 conn(0x7ff8e812ebc0 0x7ff8e812f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8edfdb700 1 --2- 172.21.15.7:0/812081822 >> v2:172.21.15.50:3300/0 conn(0x7ff8e8098540 0x7ff8e81329e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8edfdb700 1 --2- 172.21.15.7:0/812081822 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8e8098910 0x7ff8e8098ce0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7ff8edfdb700 1 -- 172.21.15.7:0/812081822 >> 172.21.15.7:0/812081822 conn(0x7ff8e8071830 msgr2=0x7ff8e80a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:50.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7f21f55e0700 1 -- start start 2023-09-30T01:54:50.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.111+0000 7f21f55e0700 10 monclient: init 2023-09-30T01:54:50.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7ff8edfdb700 1 -- 172.21.15.7:0/812081822 shutdown_connections 2023-09-30T01:54:50.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7ff8edfdb700 1 -- 172.21.15.7:0/812081822 wait complete. 2023-09-30T01:54:50.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7ff8edfdb700 10 monclient: build_initial_monmap 2023-09-30T01:54:50.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7ff8edfdb700 10 monclient: monmap: 2023-09-30T01:54:50.133 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:50.133 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:50.133 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:50.133 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:50.133 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:50.134 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.134 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:50.134 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:50.134 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:50.134 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21f55e0700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:50.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21f55e0700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:54:50.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21f55e0700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f21f0096a50 0x7f21f012fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21f55e0700 10 monclient(hunting): picked mon.noname-c con 0x7f21f0096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:50.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21f55e0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21f0096e20 0x7f21f009e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21f55e0700 10 monclient(hunting): picked mon.noname-a con 0x7f21f0096e20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:50.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21f55e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21f0137710 0x7f21f009ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21f55e0700 10 monclient(hunting): picked mon.noname-b con 0x7f21f0137710 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:50.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21f55e0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:50.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21f55e0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f01334e0 con 0x7f21f0096e20 2023-09-30T01:54:50.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21f55e0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:50.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21f55e0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f0131c50 con 0x7f21f0137710 2023-09-30T01:54:50.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21f55e0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:50.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21f55e0700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f00a1d80 con 0x7f21f0096a50 2023-09-30T01:54:50.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21f55e0700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:50.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21f55e0700 10 monclient(hunting): authenticate will time out at 1179.177738s 2023-09-30T01:54:50.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21ee7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21f0096e20 0x7f21f009e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21ee7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21f0096e20 0x7f21f009e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54882/0 (socket says 172.21.15.7:54882) 2023-09-30T01:54:50.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21ee7fc700 1 -- 172.21.15.7:0/2226236440 learned_addr learned my addr 172.21.15.7:0/2226236440 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:50.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21ee7fc700 10 monclient(hunting): get_auth_request con 0x7f21f0096e20 auth_method 0 2023-09-30T01:54:50.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21ee7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:50.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21ee7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:50.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.112+0000 7f21ee7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:50.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7ff8edfdb700 1 Processor -- start 2023-09-30T01:54:50.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21ee7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:50.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21ee7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:50.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21ee7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:50.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21eeffd700 1 --2- 172.21.15.7:0/2226236440 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f21f0096a50 0x7f21f012fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21eeffd700 10 monclient(hunting): get_auth_request con 0x7f21f0096a50 auth_method 0 2023-09-30T01:54:50.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21ef7fe700 1 --2- 172.21.15.7:0/2226236440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21f0137710 0x7f21f009ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21eeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:50.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21eeffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:50.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21eeffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:50.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21ef7fe700 10 monclient(hunting): get_auth_request con 0x7f21f0137710 auth_method 0 2023-09-30T01:54:50.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21ef7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:50.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21ef7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:50.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21ef7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:50.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21ee7fc700 10 monclient(hunting): handle_auth_done global_id 4287 payload 824 2023-09-30T01:54:50.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21ee7fc700 1 -- 172.21.15.7:0/2226236440 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f21f0096a50 msgr2=0x7f21f012fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21ee7fc700 1 --2- 172.21.15.7:0/2226236440 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f21f0096a50 0x7f21f012fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21ee7fc700 1 -- 172.21.15.7:0/2226236440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21f0137710 msgr2=0x7f21f009ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21ee7fc700 1 --2- 172.21.15.7:0/2226236440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21f0137710 0x7f21f009ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21ee7fc700 10 monclient: _finish_hunting 0 2023-09-30T01:54:50.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21ee7fc700 1 monclient: found mon.noname-a 2023-09-30T01:54:50.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21ee7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:50.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21ee7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:50.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7f21ee7fc700 1 -- 172.21.15.7:0/2226236440 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e4009420 con 0x7f21f0096e20 2023-09-30T01:54:50.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7ff8edfdb700 1 -- start start 2023-09-30T01:54:50.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7ff8edfdb700 10 monclient: init 2023-09-30T01:54:50.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7ff8edfdb700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:50.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7ff8edfdb700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:54:50.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7ff8edfdb700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8e8098540 0x7ff8e809e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7ff8edfdb700 10 monclient(hunting): picked mon.noname-c con 0x7ff8e8098540 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:50.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7ff8edfdb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8e8098910 0x7ff8e812de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7ff8edfdb700 10 monclient(hunting): picked mon.noname-a con 0x7ff8e8098910 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:50.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7ff8edfdb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8e812ebc0 0x7ff8e81414b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7ff8edfdb700 10 monclient(hunting): picked mon.noname-b con 0x7ff8e812ebc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:50.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7ff8edfdb700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:50.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7ff8edfdb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e809be40 con 0x7ff8e8098910 2023-09-30T01:54:50.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7ff8edfdb700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:50.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7ff8edfdb700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e809f3a0 con 0x7ff8e812ebc0 2023-09-30T01:54:50.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7ff8edfdb700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:50.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7ff8edfdb700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e809bb60 con 0x7ff8e8098540 2023-09-30T01:54:50.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7ff8edfdb700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:50.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.113+0000 7ff8edfdb700 10 monclient(hunting): authenticate will time out at 1179.178964s 2023-09-30T01:54:50.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ecfd9700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8e8098540 0x7ff8e809e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ecfd9700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8e8098540 0x7ff8e809e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59554/0 (socket says 172.21.15.7:59554) 2023-09-30T01:54:50.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ecfd9700 1 -- 172.21.15.7:0/3008280191 learned_addr learned my addr 172.21.15.7:0/3008280191 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:50.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ecfd9700 10 monclient(hunting): get_auth_request con 0x7ff8e8098540 auth_method 0 2023-09-30T01:54:50.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ecfd9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:50.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ecfd9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:50.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ecfd9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:50.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ecfd9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:50.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ecfd9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:50.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ecfd9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:50.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ed7da700 1 --2- 172.21.15.7:0/3008280191 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8e812ebc0 0x7ff8e81414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ed7da700 10 monclient(hunting): get_auth_request con 0x7ff8e812ebc0 auth_method 0 2023-09-30T01:54:50.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ed7da700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:50.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ed7da700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:50.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ed7da700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:50.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ed7da700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:50.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ed7da700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:50.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ed7da700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:50.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ecfd9700 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2023-09-30T01:54:50.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ecfd9700 1 -- 172.21.15.7:0/3008280191 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8e812ebc0 msgr2=0x7ff8e81414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ecfd9700 1 --2- 172.21.15.7:0/3008280191 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8e812ebc0 0x7ff8e81414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ecfd9700 1 -- 172.21.15.7:0/3008280191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8e8098910 msgr2=0x7ff8e812de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ecfd9700 1 --2- 172.21.15.7:0/3008280191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8e8098910 0x7ff8e812de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ecfd9700 10 monclient: _finish_hunting 0 2023-09-30T01:54:50.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ecfd9700 1 monclient: found mon.noname-c 2023-09-30T01:54:50.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ecfd9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:50.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ecfd9700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:54:50.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.114+0000 7ff8ecfd9700 1 -- 172.21.15.7:0/3008280191 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8dc009420 con 0x7ff8e8098540 2023-09-30T01:54:50.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.115+0000 7f21ee7fc700 1 -- 172.21.15.7:0/2226236440 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21f0096e20 msgr2=0x7f21f009e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:50.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.115+0000 7f21ee7fc700 1 --2- 172.21.15.7:0/2226236440 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21f0096e20 0x7f21f009e800 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f21f00a2b60 tx=0x7f21e4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:50.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.115+0000 7f21d6ffd700 1 -- 172.21.15.7:0/2226236440 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e4006af0 con 0x7f21f0096e20 2023-09-30T01:54:50.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.115+0000 7f21d6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:50.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.116+0000 7f21d6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:50.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.116+0000 7f21d6ffd700 10 monclient: dump: 2023-09-30T01:54:50.148 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:50.148 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:50.148 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.148 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.148 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:50.148 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.148 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:50.149 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:50.149 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:50.149 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.116+0000 7f21d6ffd700 10 monclient: _finish_auth 0 2023-09-30T01:54:50.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.116+0000 7f21d6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:50.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.116+0000 7f21f55e0700 5 monclient: authenticate success, global_id 4287 2023-09-30T01:54:50.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.116+0000 7f21d6ffd700 1 -- 172.21.15.7:0/2226236440 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4011420 con 0x7f21f0096e20 2023-09-30T01:54:50.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.116+0000 7f21d6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:50.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.116+0000 7f21d6ffd700 1 -- 172.21.15.7:0/2226236440 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e401b8d0 con 0x7f21f0096e20 2023-09-30T01:54:50.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.116+0000 7f21d6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:50.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.116+0000 7f21d6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:50.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.116+0000 7f21d6ffd700 10 monclient: dump: 2023-09-30T01:54:50.150 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:50.150 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:50.150 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.150 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.150 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:50.150 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.150 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:50.150 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:50.150 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:50.150 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.116+0000 7f21ed7fa700 4 set_mon_vals no callback set 2023-09-30T01:54:50.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.116+0000 7ff8ecfd9700 1 -- 172.21.15.7:0/3008280191 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8e8098540 msgr2=0x7ff8e809e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:50.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.116+0000 7ff8ecfd9700 1 --2- 172.21.15.7:0/3008280191 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8e8098540 0x7ff8e809e600 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc00ece0 tx=0x7ff8dc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:50.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.117+0000 7f21f55e0700 10 monclient: _renew_subs 2023-09-30T01:54:50.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.117+0000 7f21f55e0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:50.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.117+0000 7f21f55e0700 1 -- 172.21.15.7:0/2226236440 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21f00a0e20 con 0x7f21f0096e20 2023-09-30T01:54:50.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.117+0000 7f21f55e0700 10 monclient: _renew_subs 2023-09-30T01:54:50.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.117+0000 7f21f55e0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:50.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.117+0000 7f21f55e0700 1 -- 172.21.15.7:0/2226236440 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21f00a0220 con 0x7f21f0096e20 2023-09-30T01:54:50.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.118+0000 7f21f55e0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:50.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.118+0000 7f21f55e0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:50.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.118+0000 7f21f55e0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:50.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.118+0000 7f21f55e0700 1 -- 172.21.15.7:0/2226236440 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21f0064c10 con 0x7f21f0096e20 2023-09-30T01:54:50.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.118+0000 7ff8e4ff9700 1 -- 172.21.15.7:0/3008280191 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc00cb60 con 0x7ff8e8098540 2023-09-30T01:54:50.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.118+0000 7ff8e4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:50.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.118+0000 7ff8e4ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:54:50.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.118+0000 7ff8e4ff9700 10 monclient: dump: 2023-09-30T01:54:50.153 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:50.153 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:50.153 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.153 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.153 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:50.153 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.153 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:50.153 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:50.153 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:50.154 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.118+0000 7ff8e4ff9700 10 monclient: _finish_auth 0 2023-09-30T01:54:50.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.118+0000 7ff8e4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:50.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.119+0000 7ff8e4ff9700 1 -- 172.21.15.7:0/3008280191 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8dc00ccc0 con 0x7ff8e8098540 2023-09-30T01:54:50.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.119+0000 7ff8e4ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:50.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.119+0000 7ff8e4ff9700 1 -- 172.21.15.7:0/3008280191 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc013470 con 0x7ff8e8098540 2023-09-30T01:54:50.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.119+0000 7ff8e4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:50.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.119+0000 7ff8e4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:54:50.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.119+0000 7ff8e4ff9700 10 monclient: dump: 2023-09-30T01:54:50.154 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:50.154 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:50.154 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.155 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.155 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:50.155 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.155 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:50.155 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:50.155 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:50.155 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.119+0000 7ff8e6ffd700 4 set_mon_vals no callback set 2023-09-30T01:54:50.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.119+0000 7ff8edfdb700 5 monclient: authenticate success, global_id 4295 2023-09-30T01:54:50.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.119+0000 7ff8edfdb700 10 monclient: _renew_subs 2023-09-30T01:54:50.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.119+0000 7ff8edfdb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:50.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.119+0000 7ff8edfdb700 1 -- 172.21.15.7:0/3008280191 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d40094f0 con 0x7ff8e8098540 2023-09-30T01:54:50.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.119+0000 7ff8edfdb700 10 monclient: _renew_subs 2023-09-30T01:54:50.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.119+0000 7ff8edfdb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:50.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.119+0000 7ff8edfdb700 1 -- 172.21.15.7:0/3008280191 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e80997c0 con 0x7ff8e8098540 2023-09-30T01:54:50.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.119+0000 7f21d6ffd700 1 -- 172.21.15.7:0/2226236440 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f21e401bc80 con 0x7f21f0096e20 2023-09-30T01:54:50.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.119+0000 7f21d6ffd700 1 --2- 172.21.15.7:0/2226236440 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f21dc035830 0x7f21dc037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.119+0000 7f21d6ffd700 1 -- 172.21.15.7:0/2226236440 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f21e404a140 con 0x7f21f0096e20 2023-09-30T01:54:50.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.120+0000 7ff8e4ff9700 1 -- 172.21.15.7:0/3008280191 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7ff8dc013820 con 0x7ff8e8098540 2023-09-30T01:54:50.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.120+0000 7ff8e4ff9700 1 --2- 172.21.15.7:0/3008280191 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff8cc035830 0x7ff8cc037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.120+0000 7ff8e4ff9700 1 -- 172.21.15.7:0/3008280191 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7ff8dc048090 con 0x7ff8e8098540 2023-09-30T01:54:50.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.120+0000 7ff8edfdb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:50.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.120+0000 7ff8e7fff700 1 --2- 172.21.15.7:0/3008280191 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff8cc035830 0x7ff8cc037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.120+0000 7ff8edfdb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:50.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.120+0000 7ff8edfdb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:50.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.120+0000 7ff8edfdb700 1 -- 172.21.15.7:0/3008280191 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e809c190 con 0x7ff8e8098540 2023-09-30T01:54:50.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.120+0000 7ff8e7fff700 10 monclient: get_auth_request con 0x7ff8cc035830 auth_method 0 2023-09-30T01:54:50.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.120+0000 7f21eeffd700 1 --2- 172.21.15.7:0/2226236440 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f21dc035830 0x7f21dc037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.121+0000 7f21eeffd700 10 monclient: get_auth_request con 0x7f21dc035830 auth_method 0 2023-09-30T01:54:50.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.123+0000 7ff8e7fff700 1 --2- 172.21.15.7:0/3008280191 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff8cc035830 0x7ff8cc037ce0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8132990 tx=0x7ff8d4017000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:50.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.123+0000 7f21eeffd700 1 --2- 172.21.15.7:0/2226236440 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f21dc035830 0x7f21dc037ce0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f21e0007fd0 tx=0x7f21e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:50.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.125+0000 7ff8e4ff9700 1 -- 172.21.15.7:0/3008280191 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff8dc0236b0 con 0x7ff8e8098540 2023-09-30T01:54:50.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.125+0000 7ff8e4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:50.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.125+0000 7ff8e4ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:50.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.127+0000 7f21d6ffd700 1 -- 172.21.15.7:0/2226236440 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f21e402d760 con 0x7f21f0096e20 2023-09-30T01:54:50.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.127+0000 7f21d6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:50.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.127+0000 7f21d6ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:50.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.191+0000 7f519019e700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T01:54:50.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.191+0000 7f519019e700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T01:54:50.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.191+0000 7f519019e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.191+0000 7f519019e700 1 -- 172.21.15.7:0/1877411813 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5188064c10 con 0x7f5188137710 2023-09-30T01:54:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.191+0000 7f517dffb700 1 -- 172.21.15.7:0/1877411813 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5180026070 con 0x7f5188137710 2023-09-30T01:54:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.191+0000 7f517dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T01:54:50.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.191+0000 7f517dffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:50.192 INFO:teuthology.orchestra.run.smithi007.stdout:47244640259 2023-09-30T01:54:50.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.195+0000 7f519019e700 1 -- 172.21.15.7:0/1877411813 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f516c035830 msgr2=0x7f516c037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.195+0000 7f519019e700 1 --2- 172.21.15.7:0/1877411813 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f516c035830 0x7f516c037ce0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5184003a30 tx=0x7f5184007ea0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.195+0000 7f519019e700 10 monclient: shutdown 2023-09-30T01:54:50.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.195+0000 7f519019e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:50.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.195+0000 7f519019e700 1 -- 172.21.15.7:0/1877411813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5188137710 msgr2=0x7f518809e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.195+0000 7f519019e700 1 --2- 172.21.15.7:0/1877411813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5188137710 0x7f518809e800 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5180009f10 tx=0x7f518000b040 comp rx=0 tx=0).stop 2023-09-30T01:54:50.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.195+0000 7f519019e700 1 -- 172.21.15.7:0/1877411813 shutdown_connections 2023-09-30T01:54:50.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.195+0000 7f519019e700 1 --2- 172.21.15.7:0/1877411813 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f516c035830 0x7f516c037ce0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.195+0000 7f519019e700 1 --2- 172.21.15.7:0/1877411813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5188137710 0x7f518809e800 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.195+0000 7f519019e700 1 --2- 172.21.15.7:0/1877411813 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5188096a50 0x7f518812fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.195+0000 7f519019e700 1 --2- 172.21.15.7:0/1877411813 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5188096e20 0x7f5188130170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.195+0000 7f519019e700 1 -- 172.21.15.7:0/1877411813 >> 172.21.15.7:0/1877411813 conn(0x7f5188070cf0 msgr2=0x7f51881362e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:50.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.195+0000 7f519019e700 1 -- 172.21.15.7:0/1877411813 shutdown_connections 2023-09-30T01:54:50.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.195+0000 7f519019e700 1 -- 172.21.15.7:0/1877411813 wait complete. 2023-09-30T01:54:50.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.199+0000 7f7361c74700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T01:54:50.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.199+0000 7f7361c74700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T01:54:50.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.199+0000 7f7361c74700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:50.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.199+0000 7f7361c74700 1 -- 172.21.15.7:0/3983866012 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f735c0a2440 con 0x7f735c096e20 2023-09-30T01:54:50.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.199+0000 7f73437fe700 1 -- 172.21.15.7:0/3983866012 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7350017070 con 0x7f735c096e20 2023-09-30T01:54:50.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.199+0000 7f73437fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T01:54:50.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.199+0000 7f73437fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:50.200 INFO:teuthology.orchestra.run.smithi007.stdout:47244640259 2023-09-30T01:54:50.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.201+0000 7f7361c74700 1 -- 172.21.15.7:0/3983866012 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f73480357e0 msgr2=0x7f7348037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.201+0000 7f7361c74700 1 --2- 172.21.15.7:0/3983866012 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f73480357e0 0x7f7348037c90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f734c0093a0 tx=0x7f734c016000 comp rx=0 tx=0).stop 2023-09-30T01:54:50.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.201+0000 7f7361c74700 10 monclient: shutdown 2023-09-30T01:54:50.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.201+0000 7f7361c74700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:50.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.201+0000 7f7361c74700 1 -- 172.21.15.7:0/3983866012 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f735c096e20 msgr2=0x7f735c09cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.201+0000 7f7361c74700 1 --2- 172.21.15.7:0/3983866012 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f735c096e20 0x7f735c09cb30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f735000c5e0 tx=0x7f7350004a80 comp rx=0 tx=0).stop 2023-09-30T01:54:50.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.202+0000 7f7361c74700 1 -- 172.21.15.7:0/3983866012 shutdown_connections 2023-09-30T01:54:50.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.202+0000 7f7361c74700 1 --2- 172.21.15.7:0/3983866012 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f73480357e0 0x7f7348037c90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.202+0000 7f7361c74700 1 --2- 172.21.15.7:0/3983866012 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f735c137710 0x7f735c09d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.202+0000 7f7361c74700 1 --2- 172.21.15.7:0/3983866012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f735c096a50 0x7f735c0a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.202+0000 7f7361c74700 1 --2- 172.21.15.7:0/3983866012 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f735c096e20 0x7f735c09cb30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.202+0000 7f7361c74700 1 -- 172.21.15.7:0/3983866012 >> 172.21.15.7:0/3983866012 conn(0x7f735c070cf0 msgr2=0x7f735c130650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:50.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.202+0000 7f7361c74700 1 -- 172.21.15.7:0/3983866012 shutdown_connections 2023-09-30T01:54:50.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.202+0000 7f7361c74700 1 -- 172.21.15.7:0/3983866012 wait complete. 2023-09-30T01:54:50.215 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.2 2023-09-30T01:54:50.216 DEBUG:teuthology.parallel:result is None 2023-09-30T01:54:50.216 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.1 2023-09-30T01:54:50.216 DEBUG:teuthology.parallel:result is None 2023-09-30T01:54:50.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.229+0000 7f050d20c700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T01:54:50.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.229+0000 7f050d20c700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T01:54:50.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.229+0000 7f050d20c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:50.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.229+0000 7f050d20c700 1 -- 172.21.15.7:0/386531023 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f04e8005620 con 0x7f0508096e20 2023-09-30T01:54:50.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.229+0000 7f04eeffd700 1 -- 172.21.15.7:0/386531023 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f04fc017070 con 0x7f0508096e20 2023-09-30T01:54:50.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.229+0000 7f04eeffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T01:54:50.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.229+0000 7f04eeffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:50.230 INFO:teuthology.orchestra.run.smithi007.stdout:47244640258 2023-09-30T01:54:50.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.231+0000 7f050d20c700 1 -- 172.21.15.7:0/386531023 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f04f4035830 msgr2=0x7f04f4037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.231+0000 7f050d20c700 1 --2- 172.21.15.7:0/386531023 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f04f4035830 0x7f04f4037ce0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0500009af0 tx=0x7f0500009060 comp rx=0 tx=0).stop 2023-09-30T01:54:50.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.231+0000 7f050d20c700 10 monclient: shutdown 2023-09-30T01:54:50.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.231+0000 7f050d20c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:50.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.231+0000 7f050d20c700 1 -- 172.21.15.7:0/386531023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0508096e20 msgr2=0x7f050806dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.231+0000 7f050d20c700 1 --2- 172.21.15.7:0/386531023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0508096e20 0x7f050806dcf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00c5e0 tx=0x7f04fc004a80 comp rx=0 tx=0).stop 2023-09-30T01:54:50.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.231+0000 7f050d20c700 1 -- 172.21.15.7:0/386531023 shutdown_connections 2023-09-30T01:54:50.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.231+0000 7f050d20c700 1 --2- 172.21.15.7:0/386531023 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f04f4035830 0x7f04f4037ce0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.231+0000 7f050d20c700 1 --2- 172.21.15.7:0/386531023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0508138720 0x7f050806e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.231+0000 7f050d20c700 1 --2- 172.21.15.7:0/386531023 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0508096a50 0x7f050812c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.231+0000 7f050d20c700 1 --2- 172.21.15.7:0/386531023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0508096e20 0x7f050806dcf0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.231+0000 7f050d20c700 1 -- 172.21.15.7:0/386531023 >> 172.21.15.7:0/386531023 conn(0x7f0508070cf0 msgr2=0x7f0508139ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:50.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.231+0000 7f050d20c700 1 -- 172.21.15.7:0/386531023 shutdown_connections 2023-09-30T01:54:50.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.231+0000 7f050d20c700 1 -- 172.21.15.7:0/386531023 wait complete. 2023-09-30T01:54:50.243 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.0 2023-09-30T01:54:50.243 DEBUG:teuthology.parallel:result is None 2023-09-30T01:54:50.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.326+0000 7ff8edfdb700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T01:54:50.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.327+0000 7ff8edfdb700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T01:54:50.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.327+0000 7ff8edfdb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:54:50.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.327+0000 7ff8edfdb700 1 -- 172.21.15.7:0/3008280191 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ff8e809a240 con 0x7ff8e8098540 2023-09-30T01:54:50.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.327+0000 7ff8e4ff9700 1 -- 172.21.15.7:0/3008280191 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff8dc0232f0 con 0x7ff8e8098540 2023-09-30T01:54:50.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.327+0000 7ff8e4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T01:54:50.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.327+0000 7ff8e4ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:50.328 INFO:teuthology.orchestra.run.smithi007.stdout:47244640258 2023-09-30T01:54:50.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.329+0000 7ff8edfdb700 1 -- 172.21.15.7:0/3008280191 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff8cc035830 msgr2=0x7ff8cc037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.329+0000 7ff8edfdb700 1 --2- 172.21.15.7:0/3008280191 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff8cc035830 0x7ff8cc037ce0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8132990 tx=0x7ff8d4017000 comp rx=0 tx=0).stop 2023-09-30T01:54:50.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.329+0000 7ff8edfdb700 10 monclient: shutdown 2023-09-30T01:54:50.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.329+0000 7ff8edfdb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:50.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.329+0000 7ff8edfdb700 1 -- 172.21.15.7:0/3008280191 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8e8098540 msgr2=0x7ff8e809e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.329+0000 7ff8edfdb700 1 --2- 172.21.15.7:0/3008280191 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8e8098540 0x7ff8e809e600 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc00ece0 tx=0x7ff8dc0091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.329+0000 7ff8edfdb700 1 -- 172.21.15.7:0/3008280191 shutdown_connections 2023-09-30T01:54:50.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.329+0000 7ff8edfdb700 1 --2- 172.21.15.7:0/3008280191 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff8cc035830 0x7ff8cc037ce0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.329+0000 7ff8edfdb700 1 --2- 172.21.15.7:0/3008280191 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8e812ebc0 0x7ff8e81414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.329+0000 7ff8edfdb700 1 --2- 172.21.15.7:0/3008280191 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8e8098540 0x7ff8e809e600 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.329+0000 7ff8edfdb700 1 --2- 172.21.15.7:0/3008280191 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8e8098910 0x7ff8e812de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.329+0000 7ff8edfdb700 1 -- 172.21.15.7:0/3008280191 >> 172.21.15.7:0/3008280191 conn(0x7ff8e8071830 msgr2=0x7ff8e80a3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:50.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.329+0000 7ff8edfdb700 1 -- 172.21.15.7:0/3008280191 shutdown_connections 2023-09-30T01:54:50.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.329+0000 7ff8edfdb700 1 -- 172.21.15.7:0/3008280191 wait complete. 2023-09-30T01:54:50.341 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.3 2023-09-30T01:54:50.341 DEBUG:teuthology.parallel:result is None 2023-09-30T01:54:50.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.364+0000 7f21f55e0700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T01:54:50.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.364+0000 7f21f55e0700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T01:54:50.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.364+0000 7f21f55e0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:50.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.364+0000 7f21f55e0700 1 -- 172.21.15.7:0/2226236440 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f21f0097760 con 0x7f21f0096e20 2023-09-30T01:54:50.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.364+0000 7f21d6ffd700 1 -- 172.21.15.7:0/2226236440 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f21e4033020 con 0x7f21f0096e20 2023-09-30T01:54:50.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.364+0000 7f21d6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T01:54:50.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.364+0000 7f21d6ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:50.365 INFO:teuthology.orchestra.run.smithi007.stdout:47244640259 2023-09-30T01:54:50.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.367+0000 7f21f55e0700 1 -- 172.21.15.7:0/2226236440 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f21dc035830 msgr2=0x7f21dc037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.367+0000 7f21f55e0700 1 --2- 172.21.15.7:0/2226236440 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f21dc035830 0x7f21dc037ce0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f21e0007fd0 tx=0x7f21e0009040 comp rx=0 tx=0).stop 2023-09-30T01:54:50.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.367+0000 7f21f55e0700 10 monclient: shutdown 2023-09-30T01:54:50.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.367+0000 7f21f55e0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:50.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.367+0000 7f21f55e0700 1 -- 172.21.15.7:0/2226236440 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21f0096e20 msgr2=0x7f21f009e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.367+0000 7f21f55e0700 1 --2- 172.21.15.7:0/2226236440 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21f0096e20 0x7f21f009e800 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f21f00a2b60 tx=0x7f21e4016000 comp rx=0 tx=0).stop 2023-09-30T01:54:50.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.367+0000 7f21f55e0700 1 -- 172.21.15.7:0/2226236440 shutdown_connections 2023-09-30T01:54:50.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.367+0000 7f21f55e0700 1 --2- 172.21.15.7:0/2226236440 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f21dc035830 0x7f21dc037ce0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.367+0000 7f21f55e0700 1 --2- 172.21.15.7:0/2226236440 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21f0137710 0x7f21f009ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.367+0000 7f21f55e0700 1 --2- 172.21.15.7:0/2226236440 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f21f0096a50 0x7f21f012fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.367+0000 7f21f55e0700 1 --2- 172.21.15.7:0/2226236440 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21f0096e20 0x7f21f009e800 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.367+0000 7f21f55e0700 1 -- 172.21.15.7:0/2226236440 >> 172.21.15.7:0/2226236440 conn(0x7f21f0070cf0 msgr2=0x7f21f01336f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:50.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.367+0000 7f21f55e0700 1 -- 172.21.15.7:0/2226236440 shutdown_connections 2023-09-30T01:54:50.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.367+0000 7f21f55e0700 1 -- 172.21.15.7:0/2226236440 wait complete. 2023-09-30T01:54:50.379 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.5 2023-09-30T01:54:50.379 DEBUG:teuthology.parallel:result is None 2023-09-30T01:54:50.379 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T01:54:50.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:54:50.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.537+0000 7f19335b7700 10 monclient: get_monmap_and_config 2023-09-30T01:54:50.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.537+0000 7f19335b7700 10 monclient: build_initial_monmap 2023-09-30T01:54:50.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.537+0000 7f19335b7700 10 monclient: monmap: 2023-09-30T01:54:50.538 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:50.538 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:50.538 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:50.538711+0000 2023-09-30T01:54:50.538 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:50.538711+0000 2023-09-30T01:54:50.538 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:50.538 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.539 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:50.539 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:50.539 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:50.539 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:50.539 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:50.539 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.538+0000 7f19335b7700 1 Processor -- start 2023-09-30T01:54:50.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.538+0000 7f19335b7700 1 -- start start 2023-09-30T01:54:50.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.538+0000 7f19335b7700 10 monclient: init 2023-09-30T01:54:50.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.538+0000 7f19335b7700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:50.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.538+0000 7f19335b7700 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-09-30T01:54:50.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.538+0000 7f19335b7700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f192c096a50 0x7f192c1327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.538+0000 7f19335b7700 10 monclient(hunting): picked mon.noname-b con 0x7f192c096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:50.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.538+0000 7f19335b7700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f192c0a2390 0x7f192c096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.538+0000 7f19335b7700 10 monclient(hunting): picked mon.noname-c con 0x7f192c0a2390 addr v2:172.21.15.50:3300/0 2023-09-30T01:54:50.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.538+0000 7f19335b7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f192c097360 0x7f192c12ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.538+0000 7f19335b7700 10 monclient(hunting): picked mon.noname-a con 0x7f192c097360 addr v2:172.21.15.7:3300/0 2023-09-30T01:54:50.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.538+0000 7f19335b7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:50.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.538+0000 7f19335b7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f192c0980e0 con 0x7f192c097360 2023-09-30T01:54:50.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.538+0000 7f19335b7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:50.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.538+0000 7f19335b7700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f192c0991a0 con 0x7f192c0a2390 2023-09-30T01:54:50.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.538+0000 7f19335b7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:50.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.538+0000 7f19335b7700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f192c099480 con 0x7f192c096a50 2023-09-30T01:54:50.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.538+0000 7f19335b7700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:50.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.538+0000 7f19335b7700 10 monclient(hunting): authenticate will time out at 1179.603912s 2023-09-30T01:54:50.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931b54700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f192c097360 0x7f192c12ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931353700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f192c096a50 0x7f192c1327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931353700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f192c096a50 0x7f192c1327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59558/0 (socket says 172.21.15.7:59558) 2023-09-30T01:54:50.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931b54700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f192c097360 0x7f192c12ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54904/0 (socket says 172.21.15.7:54904) 2023-09-30T01:54:50.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931353700 1 -- 172.21.15.7:0/2153458808 learned_addr learned my addr 172.21.15.7:0/2153458808 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:50.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931353700 10 monclient(hunting): get_auth_request con 0x7f192c096a50 auth_method 0 2023-09-30T01:54:50.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931353700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:50.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931353700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:50.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931353700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:50.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931b54700 10 monclient(hunting): get_auth_request con 0x7f192c097360 auth_method 0 2023-09-30T01:54:50.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931b54700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:50.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931b54700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:50.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931b54700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:50.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931b54700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:50.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931b54700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:50.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931b54700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:50.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931353700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:50.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931353700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:50.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1930b52700 1 --2- 172.21.15.7:0/2153458808 >> v2:172.21.15.50:3300/0 conn(0x7f192c0a2390 0x7f192c096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931353700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:50.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1930b52700 10 monclient(hunting): get_auth_request con 0x7f192c0a2390 auth_method 0 2023-09-30T01:54:50.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1930b52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:50.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1930b52700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:50.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1930b52700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:50.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1930b52700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:50.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1930b52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:50.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1930b52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:50.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931b54700 10 monclient(hunting): handle_auth_done global_id 4290 payload 274 2023-09-30T01:54:50.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931b54700 1 -- 172.21.15.7:0/2153458808 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f192c096a50 msgr2=0x7f192c1327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:50.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931b54700 1 --2- 172.21.15.7:0/2153458808 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f192c096a50 0x7f192c1327a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931b54700 1 -- 172.21.15.7:0/2153458808 >> v2:172.21.15.50:3300/0 conn(0x7f192c0a2390 msgr2=0x7f192c096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:50.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931b54700 1 --2- 172.21.15.7:0/2153458808 >> v2:172.21.15.50:3300/0 conn(0x7f192c0a2390 0x7f192c096e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931b54700 10 monclient: _finish_hunting 0 2023-09-30T01:54:50.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931b54700 1 monclient: found mon.noname-a 2023-09-30T01:54:50.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931b54700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:50.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931b54700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:50.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931b54700 1 -- 172.21.15.7:0/2153458808 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f192c097bf0 con 0x7f192c097360 2023-09-30T01:54:50.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.539+0000 7f1931353700 1 --2- 172.21.15.7:0/2153458808 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f192c096a50 0x7f192c1327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:50.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.541+0000 7f1931b54700 1 -- 172.21.15.7:0/2153458808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f192c097360 msgr2=0x7f192c12ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:50.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.541+0000 7f1931b54700 1 --2- 172.21.15.7:0/2153458808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f192c097360 0x7f192c12ef00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f192c06ad20 tx=0x7f19240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=568e74c6eaf0f5e8 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:50.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.541+0000 7f192bfff700 1 -- 172.21.15.7:0/2153458808 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1924011840 con 0x7f192c097360 2023-09-30T01:54:50.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.541+0000 7f192bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:50.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.541+0000 7f192bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:50.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.541+0000 7f192bfff700 10 monclient: dump: 2023-09-30T01:54:50.554 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:50.554 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:50.555 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.555 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.555 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:50.555 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.555 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:50.555 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:50.555 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:50.555 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.541+0000 7f192bfff700 10 monclient: _finish_auth 0 2023-09-30T01:54:50.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.541+0000 7f192bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:50.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.541+0000 7f19335b7700 5 monclient: authenticate success, global_id 4290 2023-09-30T01:54:50.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.541+0000 7f192bfff700 1 -- 172.21.15.7:0/2153458808 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19240119c0 con 0x7f192c097360 2023-09-30T01:54:50.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.541+0000 7f192bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:50.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.541+0000 7f192bfff700 1 -- 172.21.15.7:0/2153458808 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1924011d80 con 0x7f192c097360 2023-09-30T01:54:50.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.541+0000 7f19335b7700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:50.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.541+0000 7f19335b7700 4 set_mon_vals no callback set 2023-09-30T01:54:50.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.541+0000 7f192bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:50.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.541+0000 7f192bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:50.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.541+0000 7f192bfff700 10 monclient: dump: 2023-09-30T01:54:50.556 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:50.557 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:50.557 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.557 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.557 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:50.557 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.557 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:50.557 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:50.558 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:50.558 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.542+0000 7f19335b7700 10 monclient: shutdown 2023-09-30T01:54:50.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.542+0000 7f19335b7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:50.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.542+0000 7f19335b7700 1 -- 172.21.15.7:0/2153458808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f192c097360 msgr2=0x7f192c12ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.542+0000 7f19335b7700 1 --2- 172.21.15.7:0/2153458808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f192c097360 0x7f192c12ef00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f192c06ad20 tx=0x7f19240091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.542+0000 7f19335b7700 1 -- 172.21.15.7:0/2153458808 shutdown_connections 2023-09-30T01:54:50.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.542+0000 7f19335b7700 1 --2- 172.21.15.7:0/2153458808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f192c097360 0x7f192c12ef00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.542+0000 7f19335b7700 1 --2- 172.21.15.7:0/2153458808 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f192c096a50 0x7f192c1327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.542+0000 7f19335b7700 1 --2- 172.21.15.7:0/2153458808 >> v2:172.21.15.50:3300/0 conn(0x7f192c0a2390 0x7f192c096e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.542+0000 7f19335b7700 1 -- 172.21.15.7:0/2153458808 >> 172.21.15.7:0/2153458808 conn(0x7f192c070cf0 msgr2=0x7f192c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:50.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.542+0000 7f19335b7700 1 -- 172.21.15.7:0/2153458808 shutdown_connections 2023-09-30T01:54:50.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.542+0000 7f19335b7700 1 -- 172.21.15.7:0/2153458808 wait complete. 2023-09-30T01:54:50.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.542+0000 7f19335b7700 10 monclient: build_initial_monmap 2023-09-30T01:54:50.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.542+0000 7f19335b7700 10 monclient: monmap: 2023-09-30T01:54:50.559 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:50.559 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:50.559 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:50.560 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:50.560 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:50.560 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.560 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:50.560 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:50.560 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:50.560 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.543+0000 7f19335b7700 1 Processor -- start 2023-09-30T01:54:50.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.543+0000 7f19335b7700 1 -- start start 2023-09-30T01:54:50.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.543+0000 7f19335b7700 10 monclient: init 2023-09-30T01:54:50.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.543+0000 7f19335b7700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:50.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.543+0000 7f19335b7700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:54:50.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.543+0000 7f19335b7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f192c096a50 0x7f192c1418f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.543+0000 7f19335b7700 10 monclient(hunting): picked mon.noname-a con 0x7f192c096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:50.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.543+0000 7f19335b7700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f192c097360 0x7f192c141e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.543+0000 7f19335b7700 10 monclient(hunting): picked mon.noname-c con 0x7f192c097360 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:50.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.543+0000 7f19335b7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f192c0a2390 0x7f192c0a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f19335b7700 10 monclient(hunting): picked mon.noname-b con 0x7f192c0a2390 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:50.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f19335b7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:50.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f19335b7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192c11cb40 con 0x7f192c096a50 2023-09-30T01:54:50.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f19335b7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:50.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f19335b7700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192c11e0e0 con 0x7f192c0a2390 2023-09-30T01:54:50.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f19335b7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:50.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f19335b7700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f192c11cfa0 con 0x7f192c097360 2023-09-30T01:54:50.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f19335b7700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:50.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f19335b7700 10 monclient(hunting): authenticate will time out at 1179.609033s 2023-09-30T01:54:50.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931353700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f192c096a50 0x7f192c1418f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1930b52700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f192c097360 0x7f192c141e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931353700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f192c096a50 0x7f192c1418f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54906/0 (socket says 172.21.15.7:54906) 2023-09-30T01:54:50.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1930b52700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f192c097360 0x7f192c141e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59572/0 (socket says 172.21.15.7:59572) 2023-09-30T01:54:50.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931353700 1 -- 172.21.15.7:0/3114700675 learned_addr learned my addr 172.21.15.7:0/3114700675 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:50.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931353700 10 monclient(hunting): get_auth_request con 0x7f192c096a50 auth_method 0 2023-09-30T01:54:50.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931353700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:50.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931353700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:50.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931353700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:50.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1930b52700 10 monclient(hunting): get_auth_request con 0x7f192c097360 auth_method 0 2023-09-30T01:54:50.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1930b52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:50.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1930b52700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:50.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1930b52700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:50.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931353700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:50.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931353700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:50.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931353700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:50.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1930b52700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:50.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1930b52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:50.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1930b52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:50.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931b54700 1 --2- 172.21.15.7:0/3114700675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f192c0a2390 0x7f192c0a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931353700 10 monclient(hunting): handle_auth_done global_id 4293 payload 824 2023-09-30T01:54:50.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931353700 1 -- 172.21.15.7:0/3114700675 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f192c097360 msgr2=0x7f192c141e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931353700 1 --2- 172.21.15.7:0/3114700675 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f192c097360 0x7f192c141e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931353700 1 -- 172.21.15.7:0/3114700675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f192c0a2390 msgr2=0x7f192c0a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931353700 1 --2- 172.21.15.7:0/3114700675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f192c0a2390 0x7f192c0a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931353700 10 monclient: _finish_hunting 0 2023-09-30T01:54:50.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931353700 1 monclient: found mon.noname-a 2023-09-30T01:54:50.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931353700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:50.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931353700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:50.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931353700 1 -- 172.21.15.7:0/3114700675 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f191c009ae0 con 0x7f192c096a50 2023-09-30T01:54:50.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931b54700 10 monclient: get_auth_request con 0x7f192c0a2390 auth_method 0 2023-09-30T01:54:50.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1931b54700 1 --2- 172.21.15.7:0/3114700675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f192c0a2390 0x7f192c0a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:54:50.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.544+0000 7f1930b52700 1 --2- 172.21.15.7:0/3114700675 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f192c097360 0x7f192c141e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:50.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f1931353700 1 -- 172.21.15.7:0/3114700675 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f192c096a50 msgr2=0x7f192c1418f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:50.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f1931353700 1 --2- 172.21.15.7:0/3114700675 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f192c096a50 0x7f192c1418f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f191c00c170 tx=0x7f191c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:50.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f19297fa700 1 -- 172.21.15.7:0/3114700675 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191c009dc0 con 0x7f192c096a50 2023-09-30T01:54:50.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f19297fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:50.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f19297fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:50.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f19297fa700 10 monclient: dump: 2023-09-30T01:54:50.566 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:50.566 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:50.566 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.566 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.566 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:50.566 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.566 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:50.566 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:50.566 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:50.567 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f19297fa700 10 monclient: _finish_auth 0 2023-09-30T01:54:50.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f19297fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:50.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f19297fa700 1 -- 172.21.15.7:0/3114700675 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f191c00f040 con 0x7f192c096a50 2023-09-30T01:54:50.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f19335b7700 5 monclient: authenticate success, global_id 4293 2023-09-30T01:54:50.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f19297fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:50.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f19335b7700 10 monclient: _renew_subs 2023-09-30T01:54:50.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f19297fa700 1 -- 172.21.15.7:0/3114700675 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191c014480 con 0x7f192c096a50 2023-09-30T01:54:50.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f19335b7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:50.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f19335b7700 1 -- 172.21.15.7:0/3114700675 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1924009420 con 0x7f192c096a50 2023-09-30T01:54:50.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f19297fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:50.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f19297fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:50.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f192b7fe700 4 set_mon_vals no callback set 2023-09-30T01:54:50.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f19297fa700 10 monclient: dump: 2023-09-30T01:54:50.568 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:50.568 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:50.568 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.568 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.568 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:50.568 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.568 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:50.568 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:50.568 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:50.568 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f19335b7700 10 monclient: _renew_subs 2023-09-30T01:54:50.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f19335b7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:50.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.546+0000 7f19335b7700 1 -- 172.21.15.7:0/3114700675 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f192c11c2d0 con 0x7f192c096a50 2023-09-30T01:54:50.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.547+0000 7f19335b7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:50.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.547+0000 7f19335b7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:50.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.547+0000 7f19297fa700 1 -- 172.21.15.7:0/3114700675 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f191c0147f0 con 0x7f192c096a50 2023-09-30T01:54:50.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.547+0000 7f19335b7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:50.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.547+0000 7f19335b7700 1 -- 172.21.15.7:0/3114700675 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f192c11e580 con 0x7f192c096a50 2023-09-30T01:54:50.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.548+0000 7f19297fa700 1 --2- 172.21.15.7:0/3114700675 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f19180357e0 0x7f1918037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.548+0000 7f19297fa700 1 -- 172.21.15.7:0/3114700675 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f191c048440 con 0x7f192c096a50 2023-09-30T01:54:50.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.548+0000 7f1930b52700 1 --2- 172.21.15.7:0/3114700675 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f19180357e0 0x7f1918037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.548+0000 7f1930b52700 10 monclient: get_auth_request con 0x7f19180357e0 auth_method 0 2023-09-30T01:54:50.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.550+0000 7f1930b52700 1 --2- 172.21.15.7:0/3114700675 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f19180357e0 0x7f1918037c90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f192c11c770 tx=0x7f19200091a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:50.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.552+0000 7f19297fa700 1 -- 172.21.15.7:0/3114700675 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f191c01bb40 con 0x7f192c096a50 2023-09-30T01:54:50.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.552+0000 7f19297fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:50.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.552+0000 7f19297fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:50.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.741+0000 7f19335b7700 1 -- 172.21.15.7:0/3114700675 --> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f192c064c10 con 0x7f19180357e0 2023-09-30T01:54:50.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.743+0000 7f19297fa700 1 -- 172.21.15.7:0/3114700675 <== mgr.4105 v2:172.21.15.50:6824/86200 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40408 (secure 0 0 0) 0x7f192c064c10 con 0x7f19180357e0 2023-09-30T01:54:50.744 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.745+0000 7f19335b7700 1 -- 172.21.15.7:0/3114700675 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f19180357e0 msgr2=0x7f1918037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.745+0000 7f19335b7700 1 --2- 172.21.15.7:0/3114700675 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f19180357e0 0x7f1918037c90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f192c11c770 tx=0x7f19200091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.745+0000 7f19335b7700 10 monclient: shutdown 2023-09-30T01:54:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.745+0000 7f19335b7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.745+0000 7f19335b7700 1 -- 172.21.15.7:0/3114700675 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f192c096a50 msgr2=0x7f192c1418f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.745+0000 7f19335b7700 1 --2- 172.21.15.7:0/3114700675 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f192c096a50 0x7f192c1418f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f191c00c170 tx=0x7f191c00b040 comp rx=0 tx=0).stop 2023-09-30T01:54:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.745+0000 7f19335b7700 1 -- 172.21.15.7:0/3114700675 shutdown_connections 2023-09-30T01:54:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.745+0000 7f19335b7700 1 --2- 172.21.15.7:0/3114700675 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f19180357e0 0x7f1918037c90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.745+0000 7f19335b7700 1 --2- 172.21.15.7:0/3114700675 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f192c096a50 0x7f192c1418f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.745+0000 7f19335b7700 1 --2- 172.21.15.7:0/3114700675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f192c0a2390 0x7f192c0a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.745+0000 7f19335b7700 1 --2- 172.21.15.7:0/3114700675 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f192c097360 0x7f192c141e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.745+0000 7f19335b7700 1 -- 172.21.15.7:0/3114700675 >> 172.21.15.7:0/3114700675 conn(0x7f192c070cf0 msgr2=0x7f192c0a3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.746+0000 7f19335b7700 1 -- 172.21.15.7:0/3114700675 shutdown_connections 2023-09-30T01:54:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.746+0000 7f19335b7700 1 -- 172.21.15.7:0/3114700675 wait complete. 2023-09-30T01:54:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-09-30T01:54:50.758 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-09-30T01:54:50.264805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":33496,"kb_used_data":1840,"kb_used_omap":0,"kb_used_meta":30720,"kb_avail":566197544,"statfs":{"total":579820584960,"available":579786285056,"internally_reserved":0,"allocated":1884160,"data_stored":1301669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31457280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000575"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.042899+0000","last_change":"2023-09-30T01:54:47.042422+0000","last_active":"2023-09-30T01:54:47.042899+0000","last_peered":"2023-09-30T01:54:47.042899+0000","last_clean":"2023-09-30T01:54:47.042899+0000","last_became_active":"2023-09-30T01:54:47.042183+0000","last_became_peered":"2023-09-30T01:54:47.042183+0000","last_unstale":"2023-09-30T01:54:47.042899+0000","last_undegraded":"2023-09-30T01:54:47.042899+0000","last_fullsized":"2023-09-30T01:54:47.042899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:14:57.725898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.897849+0000","last_change":"2023-09-30T01:54:46.896909+0000","last_active":"2023-09-30T01:54:46.897849+0000","last_peered":"2023-09-30T01:54:46.897849+0000","last_clean":"2023-09-30T01:54:46.897849+0000","last_became_active":"2023-09-30T01:54:46.896620+0000","last_became_peered":"2023-09-30T01:54:46.896620+0000","last_unstale":"2023-09-30T01:54:46.897849+0000","last_undegraded":"2023-09-30T01:54:46.897849+0000","last_fullsized":"2023-09-30T01:54:46.897849+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:28:25.053882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.042370+0000","last_change":"2023-09-30T01:54:47.041973+0000","last_active":"2023-09-30T01:54:47.042370+0000","last_peered":"2023-09-30T01:54:47.042370+0000","last_clean":"2023-09-30T01:54:47.042370+0000","last_became_active":"2023-09-30T01:54:47.041671+0000","last_became_peered":"2023-09-30T01:54:47.041671+0000","last_unstale":"2023-09-30T01:54:47.042370+0000","last_undegraded":"2023-09-30T01:54:47.042370+0000","last_fullsized":"2023-09-30T01:54:47.042370+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:08:29.771169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.896726+0000","last_change":"2023-09-30T01:54:46.896432+0000","last_active":"2023-09-30T01:54:46.896726+0000","last_peered":"2023-09-30T01:54:46.896726+0000","last_clean":"2023-09-30T01:54:46.896726+0000","last_became_active":"2023-09-30T01:54:46.896191+0000","last_became_peered":"2023-09-30T01:54:46.896191+0000","last_unstale":"2023-09-30T01:54:46.896726+0000","last_undegraded":"2023-09-30T01:54:46.896726+0000","last_fullsized":"2023-09-30T01:54:46.896726+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:36:51.492282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":160,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.202781+0000","last_change":"2023-09-30T01:54:45.896629+0000","last_active":"2023-09-30T01:54:47.202781+0000","last_peered":"2023-09-30T01:54:47.202781+0000","last_clean":"2023-09-30T01:54:47.202781+0000","last_became_active":"2023-09-30T01:54:45.896315+0000","last_became_peered":"2023-09-30T01:54:45.896315+0000","last_unstale":"2023-09-30T01:54:47.202781+0000","last_undegraded":"2023-09-30T01:54:47.202781+0000","last_fullsized":"2023-09-30T01:54:47.202781+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:44.377605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:44.377605+0000","last_clean_scrub_stamp":"2023-09-30T01:54:44.377605+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T13:48:02.317525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.899133+0000","last_change":"2023-09-30T01:54:46.898655+0000","last_active":"2023-09-30T01:54:46.899133+0000","last_peered":"2023-09-30T01:54:46.899133+0000","last_clean":"2023-09-30T01:54:46.899133+0000","last_became_active":"2023-09-30T01:54:46.898402+0000","last_became_peered":"2023-09-30T01:54:46.898402+0000","last_unstale":"2023-09-30T01:54:46.899133+0000","last_undegraded":"2023-09-30T01:54:46.899133+0000","last_fullsized":"2023-09-30T01:54:46.899133+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:07:41.886018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.897962+0000","last_change":"2023-09-30T01:54:46.896375+0000","last_active":"2023-09-30T01:54:46.897962+0000","last_peered":"2023-09-30T01:54:46.897962+0000","last_clean":"2023-09-30T01:54:46.897962+0000","last_became_active":"2023-09-30T01:54:46.895973+0000","last_became_peered":"2023-09-30T01:54:46.895973+0000","last_unstale":"2023-09-30T01:54:46.897962+0000","last_undegraded":"2023-09-30T01:54:46.897962+0000","last_fullsized":"2023-09-30T01:54:46.897962+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:52:22.627216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.898698+0000","last_change":"2023-09-30T01:54:46.898322+0000","last_active":"2023-09-30T01:54:46.898698+0000","last_peered":"2023-09-30T01:54:46.898698+0000","last_clean":"2023-09-30T01:54:46.898698+0000","last_became_active":"2023-09-30T01:54:46.898101+0000","last_became_peered":"2023-09-30T01:54:46.898101+0000","last_unstale":"2023-09-30T01:54:46.898698+0000","last_undegraded":"2023-09-30T01:54:46.898698+0000","last_fullsized":"2023-09-30T01:54:46.898698+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:45:31.230458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.042677+0000","last_change":"2023-09-30T01:54:47.042174+0000","last_active":"2023-09-30T01:54:47.042677+0000","last_peered":"2023-09-30T01:54:47.042677+0000","last_clean":"2023-09-30T01:54:47.042677+0000","last_became_active":"2023-09-30T01:54:47.041914+0000","last_became_peered":"2023-09-30T01:54:47.041914+0000","last_unstale":"2023-09-30T01:54:47.042677+0000","last_undegraded":"2023-09-30T01:54:47.042677+0000","last_fullsized":"2023-09-30T01:54:47.042677+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:16:46.356909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5076,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366764,"statfs":{"total":96636764160,"available":96631566336,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5064,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366776,"statfs":{"total":96636764160,"available":96631578624,"internally_reserved":0,"allocated":131072,"data_stored":38343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6540,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365300,"statfs":{"total":96636764160,"available":96630067200,"internally_reserved":0,"allocated":663552,"data_stored":567271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5140,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366700,"statfs":{"total":96636764160,"available":96631500800,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6536,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365304,"statfs":{"total":96636764160,"available":96630071296,"internally_reserved":0,"allocated":659456,"data_stored":566410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5140,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366700,"statfs":{"total":96636764160,"available":96631500800,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:54:50.759 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:54:50.877 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.877+0000 7ff287c92700 10 monclient: get_monmap_and_config 2023-09-30T01:54:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.877+0000 7ff287c92700 10 monclient: build_initial_monmap 2023-09-30T01:54:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.877+0000 7ff287c92700 10 monclient: monmap: 2023-09-30T01:54:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:50.878630+0000 2023-09-30T01:54:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:50.878630+0000 2023-09-30T01:54:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:50.879 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.879 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.878+0000 7ff287c92700 1 Processor -- start 2023-09-30T01:54:50.882 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.878+0000 7ff287c92700 1 -- start start 2023-09-30T01:54:50.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.878+0000 7ff287c92700 10 monclient: init 2023-09-30T01:54:50.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.878+0000 7ff287c92700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:50.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.878+0000 7ff287c92700 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-09-30T01:54:50.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.878+0000 7ff287c92700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff280096a50 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:50.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.878+0000 7ff287c92700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff2800a2390 addr v1:172.21.15.50:6789/0 2023-09-30T01:54:50.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.878+0000 7ff287c92700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff280097530 0x7ff28013a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.878+0000 7ff287c92700 10 monclient(hunting): picked mon.noname-a con 0x7ff280097530 addr v2:172.21.15.7:3300/0 2023-09-30T01:54:50.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.878+0000 7ff287c92700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2800980e0 con 0x7ff280096a50 2023-09-30T01:54:50.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.878+0000 7ff287c92700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff280132eb0 con 0x7ff2800a2390 2023-09-30T01:54:50.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.878+0000 7ff287c92700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:50.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.878+0000 7ff287c92700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2800991a0 con 0x7ff280097530 2023-09-30T01:54:50.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.878+0000 7ff287c92700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.878+0000 7ff287c92700 10 monclient(hunting): authenticate will time out at 1179.943846s 2023-09-30T01:54:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.878+0000 7ff28622f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff280097530 0x7ff28013a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.878+0000 7ff285a2e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff280096a50 0x7ff2801327a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58148/0 (socket says 172.21.15.7:58148) 2023-09-30T01:54:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.878+0000 7ff285a2e700 1 -- 172.21.15.7:0/3760083744 learned_addr learned my addr 172.21.15.7:0/3760083744 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:54:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.879+0000 7ff28622f700 10 monclient(hunting): get_auth_request con 0x7ff280097530 auth_method 0 2023-09-30T01:54:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.879+0000 7ff28622f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.879+0000 7ff28622f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.879+0000 7ff28622f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.879+0000 7ff28622f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.879+0000 7ff28622f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:50.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.879+0000 7ff28622f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.879+0000 7ff28622f700 10 monclient(hunting): handle_auth_done global_id 4296 payload 274 2023-09-30T01:54:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.879+0000 7ff28622f700 1 -- 172.21.15.7:0/3760083744 >> v1:172.21.15.50:6789/0 conn(0x7ff2800a2390 legacy=0x7ff280096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.879+0000 7ff28622f700 1 -- 172.21.15.7:0/3760083744 >> v1:172.21.15.7:6789/0 conn(0x7ff280096a50 legacy=0x7ff2801327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.879+0000 7ff28622f700 10 monclient: _finish_hunting 0 2023-09-30T01:54:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.879+0000 7ff28622f700 1 monclient: found mon.noname-a 2023-09-30T01:54:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.879+0000 7ff28622f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.879+0000 7ff28622f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.879+0000 7ff28622f700 1 -- 172.21.15.7:0/3760083744 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff280099480 con 0x7ff280097530 2023-09-30T01:54:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.881+0000 7ff28622f700 1 -- 172.21.15.7:0/3760083744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff280097530 msgr2=0x7ff28013a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.881+0000 7ff28622f700 1 --2- 172.21.15.7:0/3760083744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff280097530 0x7ff28013a730 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff278009e60 tx=0x7ff278009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc5bca88ab71c0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.881+0000 7ff284a2c700 1 -- 172.21.15.7:0/3760083744 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27800e070 con 0x7ff280097530 2023-09-30T01:54:50.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.881+0000 7ff284a2c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.881+0000 7ff284a2c700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.881+0000 7ff284a2c700 10 monclient: dump: 2023-09-30T01:54:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:50.886 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:50.886 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.881+0000 7ff284a2c700 10 monclient: _finish_auth 0 2023-09-30T01:54:50.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.881+0000 7ff284a2c700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:50.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.881+0000 7ff284a2c700 1 -- 172.21.15.7:0/3760083744 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff278012430 con 0x7ff280097530 2023-09-30T01:54:50.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.881+0000 7ff287c92700 5 monclient: authenticate success, global_id 4296 2023-09-30T01:54:50.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.881+0000 7ff287c92700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:50.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.881+0000 7ff284a2c700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:50.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.881+0000 7ff287c92700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:50.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.881+0000 7ff284a2c700 1 -- 172.21.15.7:0/3760083744 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27800bea0 con 0x7ff280097530 2023-09-30T01:54:50.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.881+0000 7ff287c92700 4 set_mon_vals no callback set 2023-09-30T01:54:50.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.881+0000 7ff284a2c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:50.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.881+0000 7ff284a2c700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:50.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.881+0000 7ff284a2c700 10 monclient: dump: 2023-09-30T01:54:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:50.888 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.882+0000 7ff287c92700 10 monclient: shutdown 2023-09-30T01:54:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.882+0000 7ff287c92700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.882+0000 7ff287c92700 1 -- 172.21.15.7:0/3760083744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff280097530 msgr2=0x7ff28013a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.882+0000 7ff287c92700 1 --2- 172.21.15.7:0/3760083744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff280097530 0x7ff28013a730 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff278009e60 tx=0x7ff278009780 comp rx=0 tx=0).stop 2023-09-30T01:54:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.882+0000 7ff287c92700 1 -- 172.21.15.7:0/3760083744 shutdown_connections 2023-09-30T01:54:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.882+0000 7ff287c92700 1 --2- 172.21.15.7:0/3760083744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff280097530 0x7ff28013a730 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.882+0000 7ff287c92700 1 -- 172.21.15.7:0/3760083744 >> 172.21.15.7:0/3760083744 conn(0x7ff280070cf0 msgr2=0x7ff2800710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.882+0000 7ff287c92700 1 -- 172.21.15.7:0/3760083744 shutdown_connections 2023-09-30T01:54:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.882+0000 7ff287c92700 1 -- 172.21.15.7:0/3760083744 wait complete. 2023-09-30T01:54:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.882+0000 7ff287c92700 10 monclient: build_initial_monmap 2023-09-30T01:54:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.882+0000 7ff287c92700 10 monclient: monmap: 2023-09-30T01:54:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:50.889 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:50.890 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff287c92700 1 Processor -- start 2023-09-30T01:54:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff287c92700 1 -- start start 2023-09-30T01:54:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff287c92700 10 monclient: init 2023-09-30T01:54:50.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff287c92700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff287c92700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:54:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff287c92700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff280096a50 0x7ff28009aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff287c92700 10 monclient(hunting): picked mon.noname-b con 0x7ff280096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff287c92700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff280097530 0x7ff28009af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff287c92700 10 monclient(hunting): picked mon.noname-a con 0x7ff280097530 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff287c92700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff2800a2390 0x7ff28009b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff287c92700 10 monclient(hunting): picked mon.noname-c con 0x7ff2800a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff287c92700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff287c92700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2800a5380 con 0x7ff280097530 2023-09-30T01:54:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff287c92700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff287c92700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28009e4d0 con 0x7ff280096a50 2023-09-30T01:54:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff287c92700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff287c92700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff28009df10 con 0x7ff2800a2390 2023-09-30T01:54:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff287c92700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff287c92700 10 monclient(hunting): authenticate will time out at 1179.948603s 2023-09-30T01:54:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff28522d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff280097530 0x7ff28009af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff28522d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff280097530 0x7ff28009af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54918/0 (socket says 172.21.15.7:54918) 2023-09-30T01:54:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff28522d700 1 -- 172.21.15.7:0/1284661585 learned_addr learned my addr 172.21.15.7:0/1284661585 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff28522d700 10 monclient(hunting): get_auth_request con 0x7ff280097530 auth_method 0 2023-09-30T01:54:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff28622f700 1 --2- 172.21.15.7:0/1284661585 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff2800a2390 0x7ff28009b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff28522d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff28522d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff28522d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff28622f700 10 monclient(hunting): get_auth_request con 0x7ff2800a2390 auth_method 0 2023-09-30T01:54:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff28622f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff28622f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff28622f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff28522d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff28522d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff28522d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff28622f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff28622f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff28622f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff285a2e700 1 --2- 172.21.15.7:0/1284661585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff280096a50 0x7ff28009aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff28522d700 10 monclient(hunting): handle_auth_done global_id 4299 payload 824 2023-09-30T01:54:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff28522d700 1 -- 172.21.15.7:0/1284661585 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff2800a2390 msgr2=0x7ff28009b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.883+0000 7ff28522d700 1 --2- 172.21.15.7:0/1284661585 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff2800a2390 0x7ff28009b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.884+0000 7ff28522d700 1 -- 172.21.15.7:0/1284661585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff280096a50 msgr2=0x7ff28009aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.884+0000 7ff28522d700 1 --2- 172.21.15.7:0/1284661585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff280096a50 0x7ff28009aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.884+0000 7ff28522d700 10 monclient: _finish_hunting 0 2023-09-30T01:54:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.884+0000 7ff28522d700 1 monclient: found mon.noname-a 2023-09-30T01:54:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.884+0000 7ff28522d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.884+0000 7ff28522d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.884+0000 7ff28522d700 1 -- 172.21.15.7:0/1284661585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff278009a20 con 0x7ff280097530 2023-09-30T01:54:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.884+0000 7ff285a2e700 10 monclient: get_auth_request con 0x7ff280096a50 auth_method 0 2023-09-30T01:54:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.884+0000 7ff28622f700 1 --2- 172.21.15.7:0/1284661585 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff2800a2390 0x7ff28009b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.884+0000 7ff285a2e700 1 --2- 172.21.15.7:0/1284661585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff280096a50 0x7ff28009aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:54:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff28522d700 1 -- 172.21.15.7:0/1284661585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff280097530 msgr2=0x7ff28009af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff28522d700 1 --2- 172.21.15.7:0/1284661585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff280097530 0x7ff28009af40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff27000c5d0 tx=0x7ff270007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff275ffb700 1 -- 172.21.15.7:0/1284661585 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2700117a0 con 0x7ff280097530 2023-09-30T01:54:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff275ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff275ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff275ffb700 10 monclient: dump: 2023-09-30T01:54:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:50.899 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:50.900 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff275ffb700 10 monclient: _finish_auth 0 2023-09-30T01:54:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff275ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff287c92700 5 monclient: authenticate success, global_id 4299 2023-09-30T01:54:50.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff287c92700 10 monclient: _renew_subs 2023-09-30T01:54:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff275ffb700 1 -- 172.21.15.7:0/1284661585 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff270011920 con 0x7ff280097530 2023-09-30T01:54:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff287c92700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff287c92700 1 -- 172.21.15.7:0/1284661585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff28009e250 con 0x7ff280097530 2023-09-30T01:54:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff275ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff275ffb700 1 -- 172.21.15.7:0/1284661585 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff270011ce0 con 0x7ff280097530 2023-09-30T01:54:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff275ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff275ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff275ffb700 10 monclient: dump: 2023-09-30T01:54:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.901 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:50.902 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:50.902 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:50.902 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:50.902 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:50.902 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:50.902 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:50.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff287c92700 10 monclient: _renew_subs 2023-09-30T01:54:50.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff277fff700 4 set_mon_vals no callback set 2023-09-30T01:54:50.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff287c92700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:50.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.886+0000 7ff287c92700 1 -- 172.21.15.7:0/1284661585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff28009f130 con 0x7ff280097530 2023-09-30T01:54:50.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.887+0000 7ff275ffb700 1 -- 172.21.15.7:0/1284661585 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7ff270012630 con 0x7ff280097530 2023-09-30T01:54:50.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.887+0000 7ff287c92700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.887+0000 7ff287c92700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.887+0000 7ff287c92700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.887+0000 7ff287c92700 1 -- 172.21.15.7:0/1284661585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2800a47e0 con 0x7ff280097530 2023-09-30T01:54:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.887+0000 7ff275ffb700 1 --2- 172.21.15.7:0/1284661585 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff2640357e0 0x7ff264037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.887+0000 7ff275ffb700 1 -- 172.21.15.7:0/1284661585 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7ff270047a50 con 0x7ff280097530 2023-09-30T01:54:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.888+0000 7ff285a2e700 1 --2- 172.21.15.7:0/1284661585 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff2640357e0 0x7ff264037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.888+0000 7ff285a2e700 10 monclient: get_auth_request con 0x7ff2640357e0 auth_method 0 2023-09-30T01:54:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.890+0000 7ff285a2e700 1 --2- 172.21.15.7:0/1284661585 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff2640357e0 0x7ff264037c90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff27c005260 tx=0x7ff27c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.892+0000 7ff275ffb700 1 -- 172.21.15.7:0/1284661585 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff270011e40 con 0x7ff280097530 2023-09-30T01:54:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.892+0000 7ff275ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:50.892+0000 7ff275ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:51.080 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.080+0000 7ff287c92700 1 -- 172.21.15.7:0/1284661585 --> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff28009e640 con 0x7ff2640357e0 2023-09-30T01:54:51.082 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.082+0000 7ff275ffb700 1 -- 172.21.15.7:0/1284661585 <== mgr.4105 v2:172.21.15.50:6824/86200 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40408 (secure 0 0 0) 0x7ff28009e640 con 0x7ff2640357e0 2023-09-30T01:54:51.083 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:51.085 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.084+0000 7ff287c92700 1 -- 172.21.15.7:0/1284661585 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff2640357e0 msgr2=0x7ff264037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:51.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.084+0000 7ff287c92700 1 --2- 172.21.15.7:0/1284661585 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff2640357e0 0x7ff264037c90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff27c005260 tx=0x7ff27c009040 comp rx=0 tx=0).stop 2023-09-30T01:54:51.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.084+0000 7ff287c92700 10 monclient: shutdown 2023-09-30T01:54:51.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.084+0000 7ff287c92700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:51.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.084+0000 7ff287c92700 1 -- 172.21.15.7:0/1284661585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff280097530 msgr2=0x7ff28009af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:51.086 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.084+0000 7ff287c92700 1 --2- 172.21.15.7:0/1284661585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff280097530 0x7ff28009af40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff27000c5d0 tx=0x7ff270007f60 comp rx=0 tx=0).stop 2023-09-30T01:54:51.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.084+0000 7ff287c92700 1 -- 172.21.15.7:0/1284661585 shutdown_connections 2023-09-30T01:54:51.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.084+0000 7ff287c92700 1 --2- 172.21.15.7:0/1284661585 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff2640357e0 0x7ff264037c90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:51.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.084+0000 7ff287c92700 1 --2- 172.21.15.7:0/1284661585 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff2800a2390 0x7ff28009b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:51.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.084+0000 7ff287c92700 1 --2- 172.21.15.7:0/1284661585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff280096a50 0x7ff28009aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:51.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.084+0000 7ff287c92700 1 --2- 172.21.15.7:0/1284661585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff280097530 0x7ff28009af40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:51.087 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.084+0000 7ff287c92700 1 -- 172.21.15.7:0/1284661585 >> 172.21.15.7:0/1284661585 conn(0x7ff280070cf0 msgr2=0x7ff280131280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:51.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.084+0000 7ff287c92700 1 -- 172.21.15.7:0/1284661585 shutdown_connections 2023-09-30T01:54:51.088 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.084+0000 7ff287c92700 1 -- 172.21.15.7:0/1284661585 wait complete. 2023-09-30T01:54:51.088 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-09-30T01:54:51.096 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-09-30T01:54:50.264805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":33496,"kb_used_data":1840,"kb_used_omap":0,"kb_used_meta":30720,"kb_avail":566197544,"statfs":{"total":579820584960,"available":579786285056,"internally_reserved":0,"allocated":1884160,"data_stored":1301669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31457280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000575"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.042899+0000","last_change":"2023-09-30T01:54:47.042422+0000","last_active":"2023-09-30T01:54:47.042899+0000","last_peered":"2023-09-30T01:54:47.042899+0000","last_clean":"2023-09-30T01:54:47.042899+0000","last_became_active":"2023-09-30T01:54:47.042183+0000","last_became_peered":"2023-09-30T01:54:47.042183+0000","last_unstale":"2023-09-30T01:54:47.042899+0000","last_undegraded":"2023-09-30T01:54:47.042899+0000","last_fullsized":"2023-09-30T01:54:47.042899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:14:57.725898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.897849+0000","last_change":"2023-09-30T01:54:46.896909+0000","last_active":"2023-09-30T01:54:46.897849+0000","last_peered":"2023-09-30T01:54:46.897849+0000","last_clean":"2023-09-30T01:54:46.897849+0000","last_became_active":"2023-09-30T01:54:46.896620+0000","last_became_peered":"2023-09-30T01:54:46.896620+0000","last_unstale":"2023-09-30T01:54:46.897849+0000","last_undegraded":"2023-09-30T01:54:46.897849+0000","last_fullsized":"2023-09-30T01:54:46.897849+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:28:25.053882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.042370+0000","last_change":"2023-09-30T01:54:47.041973+0000","last_active":"2023-09-30T01:54:47.042370+0000","last_peered":"2023-09-30T01:54:47.042370+0000","last_clean":"2023-09-30T01:54:47.042370+0000","last_became_active":"2023-09-30T01:54:47.041671+0000","last_became_peered":"2023-09-30T01:54:47.041671+0000","last_unstale":"2023-09-30T01:54:47.042370+0000","last_undegraded":"2023-09-30T01:54:47.042370+0000","last_fullsized":"2023-09-30T01:54:47.042370+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:08:29.771169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.896726+0000","last_change":"2023-09-30T01:54:46.896432+0000","last_active":"2023-09-30T01:54:46.896726+0000","last_peered":"2023-09-30T01:54:46.896726+0000","last_clean":"2023-09-30T01:54:46.896726+0000","last_became_active":"2023-09-30T01:54:46.896191+0000","last_became_peered":"2023-09-30T01:54:46.896191+0000","last_unstale":"2023-09-30T01:54:46.896726+0000","last_undegraded":"2023-09-30T01:54:46.896726+0000","last_fullsized":"2023-09-30T01:54:46.896726+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:36:51.492282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":160,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.202781+0000","last_change":"2023-09-30T01:54:45.896629+0000","last_active":"2023-09-30T01:54:47.202781+0000","last_peered":"2023-09-30T01:54:47.202781+0000","last_clean":"2023-09-30T01:54:47.202781+0000","last_became_active":"2023-09-30T01:54:45.896315+0000","last_became_peered":"2023-09-30T01:54:45.896315+0000","last_unstale":"2023-09-30T01:54:47.202781+0000","last_undegraded":"2023-09-30T01:54:47.202781+0000","last_fullsized":"2023-09-30T01:54:47.202781+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:44.377605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:44.377605+0000","last_clean_scrub_stamp":"2023-09-30T01:54:44.377605+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T13:48:02.317525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.899133+0000","last_change":"2023-09-30T01:54:46.898655+0000","last_active":"2023-09-30T01:54:46.899133+0000","last_peered":"2023-09-30T01:54:46.899133+0000","last_clean":"2023-09-30T01:54:46.899133+0000","last_became_active":"2023-09-30T01:54:46.898402+0000","last_became_peered":"2023-09-30T01:54:46.898402+0000","last_unstale":"2023-09-30T01:54:46.899133+0000","last_undegraded":"2023-09-30T01:54:46.899133+0000","last_fullsized":"2023-09-30T01:54:46.899133+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:07:41.886018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.897962+0000","last_change":"2023-09-30T01:54:46.896375+0000","last_active":"2023-09-30T01:54:46.897962+0000","last_peered":"2023-09-30T01:54:46.897962+0000","last_clean":"2023-09-30T01:54:46.897962+0000","last_became_active":"2023-09-30T01:54:46.895973+0000","last_became_peered":"2023-09-30T01:54:46.895973+0000","last_unstale":"2023-09-30T01:54:46.897962+0000","last_undegraded":"2023-09-30T01:54:46.897962+0000","last_fullsized":"2023-09-30T01:54:46.897962+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:52:22.627216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.898698+0000","last_change":"2023-09-30T01:54:46.898322+0000","last_active":"2023-09-30T01:54:46.898698+0000","last_peered":"2023-09-30T01:54:46.898698+0000","last_clean":"2023-09-30T01:54:46.898698+0000","last_became_active":"2023-09-30T01:54:46.898101+0000","last_became_peered":"2023-09-30T01:54:46.898101+0000","last_unstale":"2023-09-30T01:54:46.898698+0000","last_undegraded":"2023-09-30T01:54:46.898698+0000","last_fullsized":"2023-09-30T01:54:46.898698+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:45:31.230458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.042677+0000","last_change":"2023-09-30T01:54:47.042174+0000","last_active":"2023-09-30T01:54:47.042677+0000","last_peered":"2023-09-30T01:54:47.042677+0000","last_clean":"2023-09-30T01:54:47.042677+0000","last_became_active":"2023-09-30T01:54:47.041914+0000","last_became_peered":"2023-09-30T01:54:47.041914+0000","last_unstale":"2023-09-30T01:54:47.042677+0000","last_undegraded":"2023-09-30T01:54:47.042677+0000","last_fullsized":"2023-09-30T01:54:47.042677+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:16:46.356909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5076,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366764,"statfs":{"total":96636764160,"available":96631566336,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5064,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366776,"statfs":{"total":96636764160,"available":96631578624,"internally_reserved":0,"allocated":131072,"data_stored":38343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6540,"kb_used_data":648,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365300,"statfs":{"total":96636764160,"available":96630067200,"internally_reserved":0,"allocated":663552,"data_stored":567271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5140,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366700,"statfs":{"total":96636764160,"available":96631500800,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6536,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365304,"statfs":{"total":96636764160,"available":96630071296,"internally_reserved":0,"allocated":659456,"data_stored":566410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5140,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366700,"statfs":{"total":96636764160,"available":96631500800,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:54:51.098 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T01:54:51.099 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-30T01:54:51.099 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-30T01:54:51.099 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:54:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.217+0000 7fba5ef44700 10 monclient: get_monmap_and_config 2023-09-30T01:54:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.217+0000 7fba5ef44700 10 monclient: build_initial_monmap 2023-09-30T01:54:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.217+0000 7fba5ef44700 10 monclient: monmap: 2023-09-30T01:54:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:51.218476+0000 2023-09-30T01:54:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:51.218476+0000 2023-09-30T01:54:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:51.218 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:51.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5ef44700 1 Processor -- start 2023-09-30T01:54:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5ef44700 1 -- start start 2023-09-30T01:54:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5ef44700 10 monclient: init 2023-09-30T01:54:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5ef44700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5ef44700 10 monclient: _add_conns ranks=[3,0,4,1,2] 2023-09-30T01:54:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5ef44700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba58096a50 0x7fba58132750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5ef44700 10 monclient(hunting): picked mon.noname-a con 0x7fba58096a50 addr v2:172.21.15.7:3300/0 2023-09-30T01:54:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5ef44700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fba58096e20 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5ef44700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fba581376f0 0x7fba58139ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5ef44700 10 monclient(hunting): picked mon.noname-c con 0x7fba581376f0 addr v2:172.21.15.50:3300/0 2023-09-30T01:54:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5ef44700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba58099480 con 0x7fba58096e20 2023-09-30T01:54:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5ef44700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:51.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5ef44700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba58097bf0 con 0x7fba58096a50 2023-09-30T01:54:51.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5ef44700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:51.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5ef44700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba580991a0 con 0x7fba581376f0 2023-09-30T01:54:51.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5ef44700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:51.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5ef44700 10 monclient(hunting): authenticate will time out at 1180.283663s 2023-09-30T01:54:51.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5cce0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fba58096a50 0x7fba58132750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:51.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba57fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fba58096e20 0x7fba58097240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58156/0 (socket says 172.21.15.7:58156) 2023-09-30T01:54:51.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba57fff700 1 -- 172.21.15.7:0/489365686 learned_addr learned my addr 172.21.15.7:0/489365686 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:54:51.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5cce0700 10 monclient(hunting): get_auth_request con 0x7fba58096a50 auth_method 0 2023-09-30T01:54:51.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5cce0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:51.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5cce0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:51.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5cce0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:51.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5cce0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:51.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5cce0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:51.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5cce0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:51.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.218+0000 7fba5d4e1700 1 --2- 172.21.15.7:0/489365686 >> v2:172.21.15.50:3300/0 conn(0x7fba581376f0 0x7fba58139ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:51.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.219+0000 7fba5cce0700 10 monclient(hunting): handle_auth_done global_id 4302 payload 274 2023-09-30T01:54:51.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.219+0000 7fba5cce0700 1 -- 172.21.15.7:0/489365686 >> v2:172.21.15.50:3300/0 conn(0x7fba581376f0 msgr2=0x7fba58139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:51.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.219+0000 7fba5cce0700 1 --2- 172.21.15.7:0/489365686 >> v2:172.21.15.50:3300/0 conn(0x7fba581376f0 0x7fba58139ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:51.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.219+0000 7fba5cce0700 1 -- 172.21.15.7:0/489365686 >> v1:172.21.15.7:6789/0 conn(0x7fba58096e20 legacy=0x7fba58097240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:51.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.219+0000 7fba5cce0700 10 monclient: _finish_hunting 0 2023-09-30T01:54:51.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.219+0000 7fba5cce0700 1 monclient: found mon.noname-a 2023-09-30T01:54:51.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.219+0000 7fba5cce0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:51.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.219+0000 7fba5cce0700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:51.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.219+0000 7fba5cce0700 1 -- 172.21.15.7:0/489365686 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba58099760 con 0x7fba58096a50 2023-09-30T01:54:51.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.219+0000 7fba5d4e1700 10 monclient: get_auth_request con 0x7fba581376f0 auth_method 0 2023-09-30T01:54:51.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.219+0000 7fba5d4e1700 1 --2- 172.21.15.7:0/489365686 >> v2:172.21.15.50:3300/0 conn(0x7fba581376f0 0x7fba58139ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:54:51.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba5cce0700 1 -- 172.21.15.7:0/489365686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba58096a50 msgr2=0x7fba58132750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:51.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba5cce0700 1 --2- 172.21.15.7:0/489365686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba58096a50 0x7fba58132750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fba58099ef0 tx=0x7fba4c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13d54da562647856 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:51.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba577fe700 1 -- 172.21.15.7:0/489365686 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c00b960 con 0x7fba58096a50 2023-09-30T01:54:51.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:51.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba577fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:51.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba577fe700 10 monclient: dump: 2023-09-30T01:54:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:51.227 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba577fe700 10 monclient: _finish_auth 0 2023-09-30T01:54:51.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba5ef44700 5 monclient: authenticate success, global_id 4302 2023-09-30T01:54:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba577fe700 1 -- 172.21.15.7:0/489365686 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4c00bac0 con 0x7fba58096a50 2023-09-30T01:54:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba577fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba5ef44700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba577fe700 1 -- 172.21.15.7:0/489365686 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c00be60 con 0x7fba58096a50 2023-09-30T01:54:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba5ef44700 4 set_mon_vals no callback set 2023-09-30T01:54:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba577fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba577fe700 10 monclient: dump: 2023-09-30T01:54:51.228 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:51.229 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba5ef44700 10 monclient: shutdown 2023-09-30T01:54:51.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba5ef44700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:51.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba5ef44700 1 -- 172.21.15.7:0/489365686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba58096a50 msgr2=0x7fba58132750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:51.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba5ef44700 1 --2- 172.21.15.7:0/489365686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba58096a50 0x7fba58132750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fba58099ef0 tx=0x7fba4c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:51.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba5ef44700 1 -- 172.21.15.7:0/489365686 shutdown_connections 2023-09-30T01:54:51.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba5ef44700 1 --2- 172.21.15.7:0/489365686 >> v2:172.21.15.50:3300/0 conn(0x7fba581376f0 0x7fba58139ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:51.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba5ef44700 1 --2- 172.21.15.7:0/489365686 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba58096a50 0x7fba58132750 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:51.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba5ef44700 1 -- 172.21.15.7:0/489365686 >> 172.21.15.7:0/489365686 conn(0x7fba58070cf0 msgr2=0x7fba580710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:51.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.221+0000 7fba5ef44700 1 -- 172.21.15.7:0/489365686 shutdown_connections 2023-09-30T01:54:51.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.222+0000 7fba5ef44700 1 -- 172.21.15.7:0/489365686 wait complete. 2023-09-30T01:54:51.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.222+0000 7fba5ef44700 10 monclient: build_initial_monmap 2023-09-30T01:54:51.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.222+0000 7fba5ef44700 10 monclient: monmap: 2023-09-30T01:54:51.230 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:51.230 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:51.231 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.222+0000 7fba5ef44700 1 Processor -- start 2023-09-30T01:54:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.222+0000 7fba5ef44700 1 -- start start 2023-09-30T01:54:51.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.222+0000 7fba5ef44700 10 monclient: init 2023-09-30T01:54:51.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5ef44700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:51.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5ef44700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:54:51.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5ef44700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba58096a50 0x7fba5812f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:51.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5ef44700 10 monclient(hunting): picked mon.noname-b con 0x7fba58096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:51.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5ef44700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba58096e20 0x7fba5812f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:51.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5ef44700 10 monclient(hunting): picked mon.noname-a con 0x7fba58096e20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:51.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5ef44700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fba581376f0 0x7fba5809de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:51.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5ef44700 10 monclient(hunting): picked mon.noname-c con 0x7fba581376f0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:51.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5ef44700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:51.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5ef44700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba580a1620 con 0x7fba58096e20 2023-09-30T01:54:51.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5ef44700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:51.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5ef44700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba581334c0 con 0x7fba58096a50 2023-09-30T01:54:51.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5ef44700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:51.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5ef44700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba58142970 con 0x7fba581376f0 2023-09-30T01:54:51.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5ef44700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:51.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5ef44700 10 monclient(hunting): authenticate will time out at 1180.288453s 2023-09-30T01:54:51.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba57fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba58096e20 0x7fba5812f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:51.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba57fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba58096e20 0x7fba5812f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54930/0 (socket says 172.21.15.7:54930) 2023-09-30T01:54:51.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba57fff700 1 -- 172.21.15.7:0/135714211 learned_addr learned my addr 172.21.15.7:0/135714211 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:51.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5d4e1700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fba581376f0 0x7fba5809de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:51.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba57fff700 10 monclient(hunting): get_auth_request con 0x7fba58096e20 auth_method 0 2023-09-30T01:54:51.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba57fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:51.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba57fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:51.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba57fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:51.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5d4e1700 10 monclient(hunting): get_auth_request con 0x7fba581376f0 auth_method 0 2023-09-30T01:54:51.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5d4e1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:51.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5d4e1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:51.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5d4e1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:51.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba57fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:51.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba57fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:51.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba57fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:51.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5d4e1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:51.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5d4e1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:51.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5d4e1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:51.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5cce0700 1 --2- 172.21.15.7:0/135714211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba58096a50 0x7fba5812f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:51.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5cce0700 10 monclient(hunting): get_auth_request con 0x7fba58096a50 auth_method 0 2023-09-30T01:54:51.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5cce0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:51.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5cce0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:51.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5cce0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:51.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba57fff700 10 monclient(hunting): handle_auth_done global_id 4305 payload 824 2023-09-30T01:54:51.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba57fff700 1 -- 172.21.15.7:0/135714211 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fba581376f0 msgr2=0x7fba5809de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:51.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba57fff700 1 --2- 172.21.15.7:0/135714211 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fba581376f0 0x7fba5809de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:51.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba57fff700 1 -- 172.21.15.7:0/135714211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba58096a50 msgr2=0x7fba5812f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:51.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba57fff700 1 --2- 172.21.15.7:0/135714211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba58096a50 0x7fba5812f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:51.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba57fff700 10 monclient: _finish_hunting 0 2023-09-30T01:54:51.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba57fff700 1 monclient: found mon.noname-a 2023-09-30T01:54:51.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba57fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:51.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba57fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:51.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba57fff700 1 -- 172.21.15.7:0/135714211 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4c009420 con 0x7fba58096e20 2023-09-30T01:54:51.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.223+0000 7fba5d4e1700 1 --2- 172.21.15.7:0/135714211 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fba581376f0 0x7fba5809de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:51.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba57fff700 1 -- 172.21.15.7:0/135714211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba58096e20 msgr2=0x7fba5812f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:51.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba57fff700 1 --2- 172.21.15.7:0/135714211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba58096e20 0x7fba5812f6e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fba4400c5e0 tx=0x7fba44004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:51.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba54ff9700 1 -- 172.21.15.7:0/135714211 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba44005d40 con 0x7fba58096e20 2023-09-30T01:54:51.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba54ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:51.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba54ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:51.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba54ff9700 10 monclient: dump: 2023-09-30T01:54:51.243 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:51.243 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:51.243 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:51.243 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:51.243 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:51.244 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:51.244 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:51.244 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:51.244 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:51.244 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:51.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba54ff9700 10 monclient: _finish_auth 0 2023-09-30T01:54:51.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba54ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:51.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba5ef44700 5 monclient: authenticate success, global_id 4305 2023-09-30T01:54:51.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba5ef44700 10 monclient: _renew_subs 2023-09-30T01:54:51.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba54ff9700 1 -- 172.21.15.7:0/135714211 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba440070d0 con 0x7fba58096e20 2023-09-30T01:54:51.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba5ef44700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:51.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba5ef44700 1 -- 172.21.15.7:0/135714211 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba580a0790 con 0x7fba58096e20 2023-09-30T01:54:51.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba54ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:51.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba54ff9700 1 -- 172.21.15.7:0/135714211 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba44007490 con 0x7fba58096e20 2023-09-30T01:54:51.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba54ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:51.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba54ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:51.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba54ff9700 10 monclient: dump: 2023-09-30T01:54:51.247 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:51.247 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:51.247 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:51.247 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:51.247 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:51.248 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:51.248 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:51.248 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:51.248 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:51.248 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:51.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba5ef44700 10 monclient: _renew_subs 2023-09-30T01:54:51.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba5ef44700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:51.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba5ef44700 1 -- 172.21.15.7:0/135714211 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba58137f90 con 0x7fba58096e20 2023-09-30T01:54:51.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba56ffd700 4 set_mon_vals no callback set 2023-09-30T01:54:51.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.226+0000 7fba54ff9700 1 -- 172.21.15.7:0/135714211 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fba4401b3b0 con 0x7fba58096e20 2023-09-30T01:54:51.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.227+0000 7fba5ef44700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:51.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.227+0000 7fba5ef44700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:51.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.227+0000 7fba5ef44700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:51.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.227+0000 7fba5ef44700 1 -- 172.21.15.7:0/135714211 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba5806aa30 con 0x7fba58096e20 2023-09-30T01:54:51.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.227+0000 7fba54ff9700 1 --2- 172.21.15.7:0/135714211 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fba3c0357e0 0x7fba3c037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:51.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.227+0000 7fba54ff9700 1 -- 172.21.15.7:0/135714211 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fba440472c0 con 0x7fba58096e20 2023-09-30T01:54:51.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.227+0000 7fba5cce0700 1 --2- 172.21.15.7:0/135714211 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fba3c0357e0 0x7fba3c037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:51.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.227+0000 7fba5cce0700 10 monclient: get_auth_request con 0x7fba3c0357e0 auth_method 0 2023-09-30T01:54:51.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.230+0000 7fba5cce0700 1 --2- 172.21.15.7:0/135714211 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fba3c0357e0 0x7fba3c037c90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fba4c0030c0 tx=0x7fba4c016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:51.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.231+0000 7fba54ff9700 1 -- 172.21.15.7:0/135714211 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fba44018260 con 0x7fba58096e20 2023-09-30T01:54:51.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.231+0000 7fba54ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:51.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.231+0000 7fba54ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:51.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.468+0000 7fba5ef44700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:54:51.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.468+0000 7fba5ef44700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:54:51.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.468+0000 7fba5ef44700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:51.468 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.468+0000 7fba5ef44700 1 -- 172.21.15.7:0/135714211 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fba580a1860 con 0x7fba58096e20 2023-09-30T01:54:51.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.468+0000 7fba54ff9700 1 -- 172.21.15.7:0/135714211 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fba44017100 con 0x7fba58096e20 2023-09-30T01:54:51.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.468+0000 7fba54ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:54:51.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.468+0000 7fba54ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:51.469 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:51.470 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:54:51.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.470+0000 7fba5ef44700 1 -- 172.21.15.7:0/135714211 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fba3c0357e0 msgr2=0x7fba3c037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:51.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.470+0000 7fba5ef44700 1 --2- 172.21.15.7:0/135714211 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fba3c0357e0 0x7fba3c037c90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fba4c0030c0 tx=0x7fba4c016000 comp rx=0 tx=0).stop 2023-09-30T01:54:51.471 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.470+0000 7fba5ef44700 10 monclient: shutdown 2023-09-30T01:54:51.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.470+0000 7fba5ef44700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:51.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.470+0000 7fba5ef44700 1 -- 172.21.15.7:0/135714211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba58096e20 msgr2=0x7fba5812f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:51.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.470+0000 7fba5ef44700 1 --2- 172.21.15.7:0/135714211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba58096e20 0x7fba5812f6e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fba4400c5e0 tx=0x7fba44004a80 comp rx=0 tx=0).stop 2023-09-30T01:54:51.472 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.470+0000 7fba5ef44700 1 -- 172.21.15.7:0/135714211 shutdown_connections 2023-09-30T01:54:51.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.470+0000 7fba5ef44700 1 --2- 172.21.15.7:0/135714211 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fba3c0357e0 0x7fba3c037c90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:51.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.470+0000 7fba5ef44700 1 --2- 172.21.15.7:0/135714211 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fba581376f0 0x7fba5809de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:51.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.470+0000 7fba5ef44700 1 --2- 172.21.15.7:0/135714211 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba58096a50 0x7fba5812f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:51.473 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.470+0000 7fba5ef44700 1 --2- 172.21.15.7:0/135714211 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba58096e20 0x7fba5812f6e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:51.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.470+0000 7fba5ef44700 1 -- 172.21.15.7:0/135714211 >> 172.21.15.7:0/135714211 conn(0x7fba58070cf0 msgr2=0x7fba58131700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:51.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.470+0000 7fba5ef44700 1 -- 172.21.15.7:0/135714211 shutdown_connections 2023-09-30T01:54:51.474 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:51.470+0000 7fba5ef44700 1 -- 172.21.15.7:0/135714211 wait complete. 2023-09-30T01:54:54.484 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:54:54.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.615+0000 7fa543b00700 10 monclient: get_monmap_and_config 2023-09-30T01:54:54.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.615+0000 7fa543b00700 10 monclient: build_initial_monmap 2023-09-30T01:54:54.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.615+0000 7fa543b00700 10 monclient: monmap: 2023-09-30T01:54:54.615 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:54.616576+0000 2023-09-30T01:54:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:54.616576+0000 2023-09-30T01:54:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:54.616 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:54.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.616+0000 7fa543b00700 1 Processor -- start 2023-09-30T01:54:54.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.616+0000 7fa543b00700 1 -- start start 2023-09-30T01:54:54.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.616+0000 7fa543b00700 10 monclient: init 2023-09-30T01:54:54.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.616+0000 7fa543b00700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:54.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.616+0000 7fa543b00700 10 monclient: _add_conns ranks=[0,4,3,2,1] 2023-09-30T01:54:54.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.616+0000 7fa543b00700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa53c096a50 addr v1:172.21.15.7:6789/0 2023-09-30T01:54:54.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.616+0000 7fa543b00700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa53c0a2390 0x7fa53c096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:54.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.616+0000 7fa543b00700 10 monclient(hunting): picked mon.noname-c con 0x7fa53c0a2390 addr v2:172.21.15.50:3300/0 2023-09-30T01:54:54.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.616+0000 7fa543b00700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa53c097360 0x7fa53c139720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:54.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.616+0000 7fa543b00700 10 monclient(hunting): picked mon.noname-a con 0x7fa53c097360 addr v2:172.21.15.7:3300/0 2023-09-30T01:54:54.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.616+0000 7fa543b00700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa53c0980e0 con 0x7fa53c096a50 2023-09-30T01:54:54.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.616+0000 7fa543b00700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:54.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.616+0000 7fa543b00700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa53c132eb0 con 0x7fa53c097360 2023-09-30T01:54:54.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.616+0000 7fa543b00700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:54.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.616+0000 7fa543b00700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa53c0991a0 con 0x7fa53c0a2390 2023-09-30T01:54:54.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.616+0000 7fa543b00700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:54.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.616+0000 7fa543b00700 10 monclient(hunting): authenticate will time out at 1183.681816s 2023-09-30T01:54:54.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.616+0000 7fa54209d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa53c097360 0x7fa53c139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:54.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.616+0000 7fa54189c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa53c096a50 0x7fa53c1327a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44672/0 (socket says 172.21.15.7:44672) 2023-09-30T01:54:54.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.616+0000 7fa54189c700 1 -- 172.21.15.7:0/1223228840 learned_addr learned my addr 172.21.15.7:0/1223228840 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:54:54.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54209d700 10 monclient(hunting): get_auth_request con 0x7fa53c097360 auth_method 0 2023-09-30T01:54:54.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54209d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:54.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54209d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:54.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54209d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:54.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54209d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:54.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54209d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:54.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54109b700 1 --2- 172.21.15.7:0/1223228840 >> v2:172.21.15.50:3300/0 conn(0x7fa53c0a2390 0x7fa53c096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:54.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54209d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:54.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54109b700 10 monclient(hunting): get_auth_request con 0x7fa53c0a2390 auth_method 0 2023-09-30T01:54:54.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54109b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:54.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54109b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:54.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54109b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:54.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54209d700 10 monclient(hunting): handle_auth_done global_id 4308 payload 274 2023-09-30T01:54:54.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54209d700 1 -- 172.21.15.7:0/1223228840 >> v2:172.21.15.50:3300/0 conn(0x7fa53c0a2390 msgr2=0x7fa53c096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:54.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54209d700 1 --2- 172.21.15.7:0/1223228840 >> v2:172.21.15.50:3300/0 conn(0x7fa53c0a2390 0x7fa53c096e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:54.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54209d700 1 -- 172.21.15.7:0/1223228840 >> v1:172.21.15.7:6789/0 conn(0x7fa53c096a50 legacy=0x7fa53c1327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:54.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54209d700 10 monclient: _finish_hunting 0 2023-09-30T01:54:54.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54209d700 1 monclient: found mon.noname-a 2023-09-30T01:54:54.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54209d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:54.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54209d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:54.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54209d700 1 -- 172.21.15.7:0/1223228840 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53c099480 con 0x7fa53c097360 2023-09-30T01:54:54.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.617+0000 7fa54109b700 1 --2- 172.21.15.7:0/1223228840 >> v2:172.21.15.50:3300/0 conn(0x7fa53c0a2390 0x7fa53c096e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:54:54.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.619+0000 7fa54209d700 1 -- 172.21.15.7:0/1223228840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c097360 msgr2=0x7fa53c139720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:54.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.619+0000 7fa54209d700 1 --2- 172.21.15.7:0/1223228840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c097360 0x7fa53c139720 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa53c06a070 tx=0x7fa534009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d5e22b288f9bba6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:54.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.619+0000 7fa54089a700 1 -- 172.21.15.7:0/1223228840 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53400e070 con 0x7fa53c097360 2023-09-30T01:54:54.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.619+0000 7fa54089a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:54.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.619+0000 7fa54089a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:54.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.619+0000 7fa54089a700 10 monclient: dump: 2023-09-30T01:54:54.629 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:54.629 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:54.629 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:54.629 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:54.629 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:54.629 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:54.629 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:54.629 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:54.629 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:54.630 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:54.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.619+0000 7fa54089a700 10 monclient: _finish_auth 0 2023-09-30T01:54:54.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.619+0000 7fa54089a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:54.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.619+0000 7fa54089a700 1 -- 172.21.15.7:0/1223228840 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa53400be00 con 0x7fa53c097360 2023-09-30T01:54:54.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.619+0000 7fa543b00700 5 monclient: authenticate success, global_id 4308 2023-09-30T01:54:54.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.619+0000 7fa543b00700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:54:54.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.619+0000 7fa54089a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:54.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.619+0000 7fa543b00700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:54.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.619+0000 7fa54089a700 1 -- 172.21.15.7:0/1223228840 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa534012660 con 0x7fa53c097360 2023-09-30T01:54:54.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.619+0000 7fa543b00700 4 set_mon_vals no callback set 2023-09-30T01:54:54.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.619+0000 7fa54089a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:54.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.619+0000 7fa54089a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:54.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.619+0000 7fa54089a700 10 monclient: dump: 2023-09-30T01:54:54.631 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:54.631 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:54.631 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:54.631 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:54.631 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:54.631 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:54.631 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:54.631 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:54.631 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:54.631 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:54.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.620+0000 7fa543b00700 10 monclient: shutdown 2023-09-30T01:54:54.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.620+0000 7fa543b00700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:54.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.620+0000 7fa543b00700 1 -- 172.21.15.7:0/1223228840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c097360 msgr2=0x7fa53c139720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:54.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.620+0000 7fa543b00700 1 --2- 172.21.15.7:0/1223228840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c097360 0x7fa53c139720 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa53c06a070 tx=0x7fa534009780 comp rx=0 tx=0).stop 2023-09-30T01:54:54.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.620+0000 7fa543b00700 1 -- 172.21.15.7:0/1223228840 shutdown_connections 2023-09-30T01:54:54.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.620+0000 7fa543b00700 1 --2- 172.21.15.7:0/1223228840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c097360 0x7fa53c139720 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:54.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.620+0000 7fa543b00700 1 --2- 172.21.15.7:0/1223228840 >> v2:172.21.15.50:3300/0 conn(0x7fa53c0a2390 0x7fa53c096e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:54.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.620+0000 7fa543b00700 1 -- 172.21.15.7:0/1223228840 >> 172.21.15.7:0/1223228840 conn(0x7fa53c070cf0 msgr2=0x7fa53c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:54.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.620+0000 7fa543b00700 1 -- 172.21.15.7:0/1223228840 shutdown_connections 2023-09-30T01:54:54.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.620+0000 7fa543b00700 1 -- 172.21.15.7:0/1223228840 wait complete. 2023-09-30T01:54:54.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.620+0000 7fa543b00700 10 monclient: build_initial_monmap 2023-09-30T01:54:54.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.620+0000 7fa543b00700 10 monclient: monmap: 2023-09-30T01:54:54.632 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:54.633 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:54.633 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:54.633 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:54.633 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:54.633 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:54.633 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:54.633 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:54.633 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:54.633 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:54.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa543b00700 1 Processor -- start 2023-09-30T01:54:54.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa543b00700 1 -- start start 2023-09-30T01:54:54.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa543b00700 10 monclient: init 2023-09-30T01:54:54.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa543b00700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:54.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa543b00700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:54:54.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa543b00700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa53c096a50 0x7fa53c09b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:54.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa543b00700 10 monclient(hunting): picked mon.noname-c con 0x7fa53c096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:54.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa543b00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c097360 0x7fa53c09be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:54.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa543b00700 10 monclient(hunting): picked mon.noname-a con 0x7fa53c097360 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:54.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa543b00700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa53c0a2390 0x7fa53c09c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:54.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa543b00700 10 monclient(hunting): picked mon.noname-b con 0x7fa53c0a2390 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:54.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa543b00700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:54.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa543b00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c1334e0 con 0x7fa53c097360 2023-09-30T01:54:54.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa543b00700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:54.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa543b00700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c1304f0 con 0x7fa53c0a2390 2023-09-30T01:54:54.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa543b00700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:54.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa543b00700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c09ec40 con 0x7fa53c096a50 2023-09-30T01:54:54.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa543b00700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:54.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa543b00700 10 monclient(hunting): authenticate will time out at 1183.686813s 2023-09-30T01:54:54.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa54189c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa53c096a50 0x7fa53c09b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:54.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa54109b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c097360 0x7fa53c09be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:54.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa54109b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c097360 0x7fa53c09be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59038/0 (socket says 172.21.15.7:59038) 2023-09-30T01:54:54.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa54109b700 1 -- 172.21.15.7:0/1072970410 learned_addr learned my addr 172.21.15.7:0/1072970410 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:54.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa54109b700 10 monclient(hunting): get_auth_request con 0x7fa53c097360 auth_method 0 2023-09-30T01:54:54.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa54109b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:54.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa54109b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:54.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa54109b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:54.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa54189c700 10 monclient(hunting): get_auth_request con 0x7fa53c096a50 auth_method 0 2023-09-30T01:54:54.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa54189c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:54.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa54189c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:54.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.621+0000 7fa54189c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:54.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54109b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:54.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54109b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:54.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54109b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:54.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54189c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:54.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54189c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:54.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54189c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:54.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54209d700 1 --2- 172.21.15.7:0/1072970410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa53c0a2390 0x7fa53c09c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:54.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54209d700 10 monclient(hunting): get_auth_request con 0x7fa53c0a2390 auth_method 0 2023-09-30T01:54:54.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54209d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:54.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54209d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:54.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54209d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:54.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54109b700 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2023-09-30T01:54:54.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54109b700 1 -- 172.21.15.7:0/1072970410 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa53c096a50 msgr2=0x7fa53c09b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:54.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54109b700 1 --2- 172.21.15.7:0/1072970410 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa53c096a50 0x7fa53c09b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:54.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54109b700 1 -- 172.21.15.7:0/1072970410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa53c0a2390 msgr2=0x7fa53c09c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:54.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54109b700 1 --2- 172.21.15.7:0/1072970410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa53c0a2390 0x7fa53c09c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:54.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54109b700 10 monclient: _finish_hunting 0 2023-09-30T01:54:54.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54109b700 1 monclient: found mon.noname-a 2023-09-30T01:54:54.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54109b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:54.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54109b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:54.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54109b700 1 -- 172.21.15.7:0/1072970410 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa534009a20 con 0x7fa53c097360 2023-09-30T01:54:54.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54189c700 1 --2- 172.21.15.7:0/1072970410 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa53c096a50 0x7fa53c09b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:54.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.622+0000 7fa54209d700 1 --2- 172.21.15.7:0/1072970410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa53c0a2390 0x7fa53c09c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:54:54.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa54109b700 1 -- 172.21.15.7:0/1072970410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c097360 msgr2=0x7fa53c09be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:54.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa54109b700 1 --2- 172.21.15.7:0/1072970410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c097360 0x7fa53c09be00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa52c00e900 tx=0x7fa52c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:54.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa531ffb700 1 -- 172.21.15.7:0/1072970410 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c013620 con 0x7fa53c097360 2023-09-30T01:54:54.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa531ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:54.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa531ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:54.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa531ffb700 10 monclient: dump: 2023-09-30T01:54:54.638 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:54.638 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:54.638 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:54.639 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:54.639 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:54.639 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:54.639 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:54.639 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:54.639 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:54.639 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:54.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa531ffb700 10 monclient: _finish_auth 0 2023-09-30T01:54:54.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa531ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:54.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa543b00700 5 monclient: authenticate success, global_id 4311 2023-09-30T01:54:54.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa531ffb700 1 -- 172.21.15.7:0/1072970410 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa52c013780 con 0x7fa53c097360 2023-09-30T01:54:54.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa543b00700 10 monclient: _renew_subs 2023-09-30T01:54:54.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa543b00700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:54.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa543b00700 1 -- 172.21.15.7:0/1072970410 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa53c09eef0 con 0x7fa53c097360 2023-09-30T01:54:54.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa531ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:54.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa531ffb700 1 -- 172.21.15.7:0/1072970410 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c013b20 con 0x7fa53c097360 2023-09-30T01:54:54.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa531ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:54.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa531ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:54.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa531ffb700 10 monclient: dump: 2023-09-30T01:54:54.640 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:54.640 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:54.640 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:54.640 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:54.640 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:54.640 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:54.641 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:54.641 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:54.641 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:54.641 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:54.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa533fff700 4 set_mon_vals no callback set 2023-09-30T01:54:54.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa543b00700 10 monclient: _renew_subs 2023-09-30T01:54:54.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa543b00700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:54.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.624+0000 7fa543b00700 1 -- 172.21.15.7:0/1072970410 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa53c09d3c0 con 0x7fa53c097360 2023-09-30T01:54:54.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.625+0000 7fa531ffb700 1 -- 172.21.15.7:0/1072970410 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fa52c00c400 con 0x7fa53c097360 2023-09-30T01:54:54.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.625+0000 7fa531ffb700 1 --2- 172.21.15.7:0/1072970410 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa5200357e0 0x7fa520037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:54.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.626+0000 7fa543b00700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:54.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.626+0000 7fa543b00700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:54.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.626+0000 7fa543b00700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:54.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.626+0000 7fa531ffb700 1 -- 172.21.15.7:0/1072970410 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fa52c047a40 con 0x7fa53c097360 2023-09-30T01:54:54.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.626+0000 7fa543b00700 1 -- 172.21.15.7:0/1072970410 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa53c0a4b50 con 0x7fa53c097360 2023-09-30T01:54:54.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.626+0000 7fa54189c700 1 --2- 172.21.15.7:0/1072970410 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa5200357e0 0x7fa520037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:54.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.626+0000 7fa54189c700 10 monclient: get_auth_request con 0x7fa5200357e0 auth_method 0 2023-09-30T01:54:54.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.628+0000 7fa54189c700 1 --2- 172.21.15.7:0/1072970410 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa5200357e0 0x7fa520037c90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa538005220 tx=0x7fa538007f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:54.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.630+0000 7fa531ffb700 1 -- 172.21.15.7:0/1072970410 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa52c020020 con 0x7fa53c097360 2023-09-30T01:54:54.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.630+0000 7fa531ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:54.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.630+0000 7fa531ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:54.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.867+0000 7fa543b00700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:54:54.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.867+0000 7fa543b00700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:54:54.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.867+0000 7fa543b00700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:54.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.867+0000 7fa543b00700 1 -- 172.21.15.7:0/1072970410 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa53c0a4d50 con 0x7fa53c097360 2023-09-30T01:54:54.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.867+0000 7fa531ffb700 1 -- 172.21.15.7:0/1072970410 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa52c019300 con 0x7fa53c097360 2023-09-30T01:54:54.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.867+0000 7fa531ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:54:54.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.867+0000 7fa531ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:54.868 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:54.869 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:54:54.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.869+0000 7fa543b00700 1 -- 172.21.15.7:0/1072970410 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa5200357e0 msgr2=0x7fa520037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:54.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.869+0000 7fa543b00700 1 --2- 172.21.15.7:0/1072970410 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa5200357e0 0x7fa520037c90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa538005220 tx=0x7fa538007f90 comp rx=0 tx=0).stop 2023-09-30T01:54:54.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.869+0000 7fa543b00700 10 monclient: shutdown 2023-09-30T01:54:54.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.869+0000 7fa543b00700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:54.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.870+0000 7fa543b00700 1 -- 172.21.15.7:0/1072970410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c097360 msgr2=0x7fa53c09be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:54.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.870+0000 7fa543b00700 1 --2- 172.21.15.7:0/1072970410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c097360 0x7fa53c09be00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa52c00e900 tx=0x7fa52c008040 comp rx=0 tx=0).stop 2023-09-30T01:54:54.870 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.870+0000 7fa543b00700 1 -- 172.21.15.7:0/1072970410 shutdown_connections 2023-09-30T01:54:54.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.870+0000 7fa543b00700 1 --2- 172.21.15.7:0/1072970410 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa5200357e0 0x7fa520037c90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:54.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.870+0000 7fa543b00700 1 --2- 172.21.15.7:0/1072970410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa53c0a2390 0x7fa53c09c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:54.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.870+0000 7fa543b00700 1 --2- 172.21.15.7:0/1072970410 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa53c096a50 0x7fa53c09b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:54.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.870+0000 7fa543b00700 1 --2- 172.21.15.7:0/1072970410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa53c097360 0x7fa53c09be00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:54.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.870+0000 7fa543b00700 1 -- 172.21.15.7:0/1072970410 >> 172.21.15.7:0/1072970410 conn(0x7fa53c070cf0 msgr2=0x7fa53c1313f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:54.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.870+0000 7fa543b00700 1 -- 172.21.15.7:0/1072970410 shutdown_connections 2023-09-30T01:54:54.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:54.870+0000 7fa543b00700 1 -- 172.21.15.7:0/1072970410 wait complete. 2023-09-30T01:54:57.882 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:54:58.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.012+0000 7febb08d1700 10 monclient: get_monmap_and_config 2023-09-30T01:54:58.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.012+0000 7febb08d1700 10 monclient: build_initial_monmap 2023-09-30T01:54:58.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.012+0000 7febb08d1700 10 monclient: monmap: 2023-09-30T01:54:58.013 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:58.013 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:58.013 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:58.013999+0000 2023-09-30T01:54:58.013 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:58.013999+0000 2023-09-30T01:54:58.013 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:58.013 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:58.014 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:54:58.014 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:54:58.014 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:54:58.014 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:54:58.014 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:54:58.014 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:58.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.013+0000 7febb08d1700 1 Processor -- start 2023-09-30T01:54:58.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.013+0000 7febb08d1700 1 -- start start 2023-09-30T01:54:58.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.013+0000 7febb08d1700 10 monclient: init 2023-09-30T01:54:58.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febb08d1700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:58.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febb08d1700 10 monclient: _add_conns ranks=[3,1,2,4,0] 2023-09-30T01:54:58.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febb08d1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7febac096a50 0x7febac1327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:58.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febb08d1700 10 monclient(hunting): picked mon.noname-a con 0x7febac096a50 addr v2:172.21.15.7:3300/0 2023-09-30T01:54:58.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febb08d1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7febac0a2390 addr v1:172.21.15.50:6789/0 2023-09-30T01:54:58.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febb08d1700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7febac097530 0x7febac139720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:58.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febb08d1700 10 monclient(hunting): picked mon.noname-b con 0x7febac097530 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:58.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febb08d1700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febac0980e0 con 0x7febac0a2390 2023-09-30T01:54:58.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febb08d1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:58.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febb08d1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febac0991a0 con 0x7febac096a50 2023-09-30T01:54:58.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febb08d1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:58.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febb08d1700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febac099480 con 0x7febac097530 2023-09-30T01:54:58.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febb08d1700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:58.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febb08d1700 10 monclient(hunting): authenticate will time out at 1187.079365s 2023-09-30T01:54:58.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaad9d700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7febac097530 0x7febac139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:58.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaa59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7febac096a50 0x7febac1327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:58.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaad9d700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7febac097530 0x7febac139720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33386/0 (socket says 172.21.15.7:33386) 2023-09-30T01:54:58.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaad9d700 1 -- 172.21.15.7:0/3007112875 learned_addr learned my addr 172.21.15.7:0/3007112875 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:58.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaa59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7febac096a50 0x7febac1327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59064/0 (socket says 172.21.15.7:59064) 2023-09-30T01:54:58.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaad9d700 10 monclient(hunting): get_auth_request con 0x7febac097530 auth_method 0 2023-09-30T01:54:58.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:58.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:58.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:58.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaa59c700 10 monclient(hunting): get_auth_request con 0x7febac096a50 auth_method 0 2023-09-30T01:54:58.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaa59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:58.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaa59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:58.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaa59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:58.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaa59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:58.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaa59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:58.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaa59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:58.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:58.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:58.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:58.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaa59c700 10 monclient(hunting): handle_auth_done global_id 4314 payload 274 2023-09-30T01:54:58.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaa59c700 1 -- 172.21.15.7:0/3007112875 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7febac097530 msgr2=0x7febac139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:58.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaa59c700 1 --2- 172.21.15.7:0/3007112875 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7febac097530 0x7febac139720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:58.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaa59c700 1 -- 172.21.15.7:0/3007112875 >> v1:172.21.15.50:6789/0 conn(0x7febac0a2390 legacy=0x7febac096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:54:58.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaa59c700 10 monclient: _finish_hunting 0 2023-09-30T01:54:58.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaa59c700 1 monclient: found mon.noname-a 2023-09-30T01:54:58.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaa59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:58.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaa59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:58.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaa59c700 1 -- 172.21.15.7:0/3007112875 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febac097bf0 con 0x7febac096a50 2023-09-30T01:54:58.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.014+0000 7febaad9d700 1 --2- 172.21.15.7:0/3007112875 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7febac097530 0x7febac139720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:58.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7febaa59c700 1 -- 172.21.15.7:0/3007112875 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febac096a50 msgr2=0x7febac1327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:58.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7febaa59c700 1 --2- 172.21.15.7:0/3007112875 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febac096a50 0x7febac1327a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0098e0 tx=0x7feb9c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e1f13017bb699aa server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:58.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7feba959a700 1 -- 172.21.15.7:0/3007112875 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c011840 con 0x7febac096a50 2023-09-30T01:54:58.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7feba959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:58.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7feba959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:58.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7feba959a700 10 monclient: dump: 2023-09-30T01:54:58.022 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:58.023 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:58.023 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:58.023 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:58.023 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:58.023 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:58.023 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:58.023 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:58.023 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:58.023 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:58.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7feba959a700 10 monclient: _finish_auth 0 2023-09-30T01:54:58.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7feba959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:58.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7febb08d1700 5 monclient: authenticate success, global_id 4314 2023-09-30T01:54:58.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7feba959a700 1 -- 172.21.15.7:0/3007112875 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9c0119a0 con 0x7febac096a50 2023-09-30T01:54:58.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7feba959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:58.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7febb08d1700 10 monclient: get_monmap_and_config success 2023-09-30T01:54:58.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7febb08d1700 4 set_mon_vals no callback set 2023-09-30T01:54:58.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7feba959a700 1 -- 172.21.15.7:0/3007112875 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c011d40 con 0x7febac096a50 2023-09-30T01:54:58.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7feba959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:58.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7feba959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:58.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7feba959a700 10 monclient: dump: 2023-09-30T01:54:58.024 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:58.024 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:58.024 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:58.025 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:58.025 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:58.025 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:58.025 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:58.025 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:58.025 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:58.025 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:58.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7febb08d1700 10 monclient: shutdown 2023-09-30T01:54:58.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7febb08d1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:58.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7febb08d1700 1 -- 172.21.15.7:0/3007112875 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febac096a50 msgr2=0x7febac1327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:58.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7febb08d1700 1 --2- 172.21.15.7:0/3007112875 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febac096a50 0x7febac1327a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0098e0 tx=0x7feb9c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:54:58.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7febb08d1700 1 -- 172.21.15.7:0/3007112875 shutdown_connections 2023-09-30T01:54:58.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7febb08d1700 1 --2- 172.21.15.7:0/3007112875 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7febac097530 0x7febac139720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:58.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7febb08d1700 1 --2- 172.21.15.7:0/3007112875 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febac096a50 0x7febac1327a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:58.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7febb08d1700 1 -- 172.21.15.7:0/3007112875 >> 172.21.15.7:0/3007112875 conn(0x7febac070cf0 msgr2=0x7febac0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:58.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7febb08d1700 1 -- 172.21.15.7:0/3007112875 shutdown_connections 2023-09-30T01:54:58.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.017+0000 7febb08d1700 1 -- 172.21.15.7:0/3007112875 wait complete. 2023-09-30T01:54:58.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.018+0000 7febb08d1700 10 monclient: build_initial_monmap 2023-09-30T01:54:58.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.018+0000 7febb08d1700 10 monclient: monmap: 2023-09-30T01:54:58.026 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:54:58.026 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:54:58.026 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:54:58.026 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:54:58.026 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:54:58.027 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:58.027 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:54:58.027 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:54:58.027 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:54:58.027 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:58.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.018+0000 7febb08d1700 1 Processor -- start 2023-09-30T01:54:58.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.018+0000 7febb08d1700 1 -- start start 2023-09-30T01:54:58.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.018+0000 7febb08d1700 10 monclient: init 2023-09-30T01:54:58.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febb08d1700 10 monclient: _reopen_session rank -1 2023-09-30T01:54:58.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febb08d1700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:54:58.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febb08d1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febac096a50 0x7febac141f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:58.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febb08d1700 10 monclient(hunting): picked mon.noname-a con 0x7febac096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:54:58.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febb08d1700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7febac097530 0x7febac142490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:58.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febb08d1700 10 monclient(hunting): picked mon.noname-c con 0x7febac097530 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:54:58.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febb08d1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febac0a2390 0x7febac1429d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:58.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febb08d1700 10 monclient(hunting): picked mon.noname-b con 0x7febac0a2390 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:54:58.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febb08d1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:58.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febb08d1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac130120 con 0x7febac096a50 2023-09-30T01:54:58.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febb08d1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:58.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febb08d1700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac1334e0 con 0x7febac0a2390 2023-09-30T01:54:58.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febb08d1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:54:58.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febb08d1700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febac09daf0 con 0x7febac097530 2023-09-30T01:54:58.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febb08d1700 10 monclient(hunting): _renew_subs 2023-09-30T01:54:58.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febb08d1700 10 monclient(hunting): authenticate will time out at 1187.084313s 2023-09-30T01:54:58.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaa59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febac096a50 0x7febac141f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:58.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaa59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febac096a50 0x7febac141f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59076/0 (socket says 172.21.15.7:59076) 2023-09-30T01:54:58.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaa59c700 1 -- 172.21.15.7:0/581850145 learned_addr learned my addr 172.21.15.7:0/581850145 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:54:58.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7feba9d9b700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7febac097530 0x7febac142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:58.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaa59c700 10 monclient(hunting): get_auth_request con 0x7febac096a50 auth_method 0 2023-09-30T01:54:58.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaa59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:58.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaa59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:58.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaa59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:58.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7feba9d9b700 10 monclient(hunting): get_auth_request con 0x7febac097530 auth_method 0 2023-09-30T01:54:58.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7feba9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:54:58.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7feba9d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:54:58.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7feba9d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:54:58.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaa59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:58.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaa59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:58.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaa59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:58.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7feba9d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:54:58.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7feba9d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:54:58.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7feba9d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:54:58.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaa59c700 10 monclient(hunting): handle_auth_done global_id 4317 payload 824 2023-09-30T01:54:58.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaad9d700 1 --2- 172.21.15.7:0/581850145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febac0a2390 0x7febac1429d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:58.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaa59c700 1 -- 172.21.15.7:0/581850145 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7febac097530 msgr2=0x7febac142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:58.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaa59c700 1 --2- 172.21.15.7:0/581850145 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7febac097530 0x7febac142490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:58.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaa59c700 1 -- 172.21.15.7:0/581850145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febac0a2390 msgr2=0x7febac1429d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:58.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaa59c700 1 --2- 172.21.15.7:0/581850145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febac0a2390 0x7febac1429d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:58.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaa59c700 10 monclient: _finish_hunting 0 2023-09-30T01:54:58.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaa59c700 1 monclient: found mon.noname-a 2023-09-30T01:54:58.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaa59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:54:58.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaa59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:54:58.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaa59c700 1 -- 172.21.15.7:0/581850145 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba4009ae0 con 0x7febac096a50 2023-09-30T01:54:58.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaad9d700 10 monclient: get_auth_request con 0x7febac0a2390 auth_method 0 2023-09-30T01:54:58.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7febaad9d700 1 --2- 172.21.15.7:0/581850145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febac0a2390 0x7febac1429d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:54:58.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.019+0000 7feba9d9b700 1 --2- 172.21.15.7:0/581850145 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7febac097530 0x7febac142490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:54:58.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.021+0000 7febaa59c700 1 -- 172.21.15.7:0/581850145 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febac096a50 msgr2=0x7febac141f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:54:58.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.021+0000 7febaa59c700 1 --2- 172.21.15.7:0/581850145 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febac096a50 0x7febac141f50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0093f0 tx=0x7feb9c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:58.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.022+0000 7feb9a7fc700 1 -- 172.21.15.7:0/581850145 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c006af0 con 0x7febac096a50 2023-09-30T01:54:58.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.022+0000 7feb9a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:58.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.022+0000 7feb9a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:54:58.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.022+0000 7feb9a7fc700 10 monclient: dump: 2023-09-30T01:54:58.036 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:58.036 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:58.037 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:58.037 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:58.037 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:58.037 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:58.037 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:58.037 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:58.037 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:58.037 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:58.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.022+0000 7feb9a7fc700 10 monclient: _finish_auth 0 2023-09-30T01:54:58.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.022+0000 7feb9a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:54:58.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.022+0000 7feb9a7fc700 1 -- 172.21.15.7:0/581850145 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9c020070 con 0x7febac096a50 2023-09-30T01:54:58.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.022+0000 7febb08d1700 5 monclient: authenticate success, global_id 4317 2023-09-30T01:54:58.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.022+0000 7febb08d1700 10 monclient: _renew_subs 2023-09-30T01:54:58.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.022+0000 7febb08d1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:58.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.022+0000 7febb08d1700 1 -- 172.21.15.7:0/581850145 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb9c009420 con 0x7febac096a50 2023-09-30T01:54:58.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.022+0000 7feb9a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:54:58.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.022+0000 7feb9a7fc700 1 -- 172.21.15.7:0/581850145 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c00bdd0 con 0x7febac096a50 2023-09-30T01:54:58.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.022+0000 7feb9a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:54:58.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.022+0000 7feb9a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:54:58.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.022+0000 7feb9a7fc700 10 monclient: dump: 2023-09-30T01:54:58.038 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:54:58.038 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:54:58.038 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:58.039 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:54:58.039 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:54:58.039 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:54:58.039 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:54:58.039 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:54:58.039 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:54:58.039 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:54:58.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.022+0000 7feba8d99700 4 set_mon_vals no callback set 2023-09-30T01:54:58.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.022+0000 7febb08d1700 10 monclient: _renew_subs 2023-09-30T01:54:58.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.022+0000 7febb08d1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:58.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.022+0000 7febb08d1700 1 -- 172.21.15.7:0/581850145 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febac09e1b0 con 0x7febac096a50 2023-09-30T01:54:58.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.022+0000 7feb9a7fc700 1 -- 172.21.15.7:0/581850145 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7feb9c022030 con 0x7febac096a50 2023-09-30T01:54:58.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.023+0000 7feb9a7fc700 1 --2- 172.21.15.7:0/581850145 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7feb8c0357e0 0x7feb8c037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:54:58.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.023+0000 7feb9a7fc700 1 -- 172.21.15.7:0/581850145 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7feb9c0498f0 con 0x7febac096a50 2023-09-30T01:54:58.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.023+0000 7febb08d1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:58.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.023+0000 7febb08d1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:58.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.023+0000 7febb08d1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:58.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.023+0000 7febb08d1700 1 -- 172.21.15.7:0/581850145 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febac0a5e60 con 0x7febac096a50 2023-09-30T01:54:58.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.023+0000 7feba9d9b700 1 --2- 172.21.15.7:0/581850145 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7feb8c0357e0 0x7feb8c037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:54:58.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.023+0000 7feba9d9b700 10 monclient: get_auth_request con 0x7feb8c0357e0 auth_method 0 2023-09-30T01:54:58.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.025+0000 7feba9d9b700 1 --2- 172.21.15.7:0/581850145 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7feb8c0357e0 0x7feb8c037c90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7febac09d250 tx=0x7feba0007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:54:58.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.028+0000 7feb9a7fc700 1 -- 172.21.15.7:0/581850145 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feb9c01b660 con 0x7febac096a50 2023-09-30T01:54:58.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.028+0000 7feb9a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:54:58.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.028+0000 7feb9a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:54:58.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.268+0000 7febb08d1700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:54:58.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.268+0000 7febb08d1700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:54:58.268 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.268+0000 7febb08d1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:54:58.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.268+0000 7febb08d1700 1 -- 172.21.15.7:0/581850145 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7febac0a5e60 con 0x7febac096a50 2023-09-30T01:54:58.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.268+0000 7feb9a7fc700 1 -- 172.21.15.7:0/581850145 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7feb9c0193f0 con 0x7febac096a50 2023-09-30T01:54:58.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.268+0000 7feb9a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:54:58.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.268+0000 7feb9a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:54:58.270 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:54:58.270 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:54:58.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.270+0000 7febb08d1700 1 -- 172.21.15.7:0/581850145 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7feb8c0357e0 msgr2=0x7feb8c037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:58.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.270+0000 7febb08d1700 1 --2- 172.21.15.7:0/581850145 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7feb8c0357e0 0x7feb8c037c90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7febac09d250 tx=0x7feba0007ea0 comp rx=0 tx=0).stop 2023-09-30T01:54:58.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.270+0000 7febb08d1700 10 monclient: shutdown 2023-09-30T01:54:58.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.270+0000 7febb08d1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:54:58.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.270+0000 7febb08d1700 1 -- 172.21.15.7:0/581850145 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febac096a50 msgr2=0x7febac141f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:54:58.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.270+0000 7febb08d1700 1 --2- 172.21.15.7:0/581850145 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febac096a50 0x7febac141f50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feb9c0093f0 tx=0x7feb9c016000 comp rx=0 tx=0).stop 2023-09-30T01:54:58.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.270+0000 7febb08d1700 1 -- 172.21.15.7:0/581850145 shutdown_connections 2023-09-30T01:54:58.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.270+0000 7febb08d1700 1 --2- 172.21.15.7:0/581850145 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7feb8c0357e0 0x7feb8c037c90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:58.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.270+0000 7febb08d1700 1 --2- 172.21.15.7:0/581850145 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7febac096a50 0x7febac141f50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:58.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.270+0000 7febb08d1700 1 --2- 172.21.15.7:0/581850145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7febac0a2390 0x7febac1429d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:58.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.270+0000 7febb08d1700 1 --2- 172.21.15.7:0/581850145 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7febac097530 0x7febac142490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:54:58.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.270+0000 7febb08d1700 1 -- 172.21.15.7:0/581850145 >> 172.21.15.7:0/581850145 conn(0x7febac070cf0 msgr2=0x7febac0a44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:54:58.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.270+0000 7febb08d1700 1 -- 172.21.15.7:0/581850145 shutdown_connections 2023-09-30T01:54:58.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:54:58.270+0000 7febb08d1700 1 -- 172.21.15.7:0/581850145 wait complete. 2023-09-30T01:55:01.283 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:55:01.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.413+0000 7ff8fa6da700 10 monclient: get_monmap_and_config 2023-09-30T01:55:01.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.413+0000 7ff8fa6da700 10 monclient: build_initial_monmap 2023-09-30T01:55:01.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.414+0000 7ff8fa6da700 10 monclient: monmap: 2023-09-30T01:55:01.414 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:01.414 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:01.414 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:55:01.415037+0000 2023-09-30T01:55:01.415 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:55:01.415037+0000 2023-09-30T01:55:01.415 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:01.415 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:01.415 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:55:01.415 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:55:01.415 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:55:01.415 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:55:01.415 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:55:01.415 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:01.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.414+0000 7ff8fa6da700 1 Processor -- start 2023-09-30T01:55:01.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.414+0000 7ff8fa6da700 1 -- start start 2023-09-30T01:55:01.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.414+0000 7ff8fa6da700 10 monclient: init 2023-09-30T01:55:01.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8fa6da700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:01.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8fa6da700 10 monclient: _add_conns ranks=[2,4,1,0,3] 2023-09-30T01:55:01.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8fa6da700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8f4096a50 0x7ff8f41327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:01.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8fa6da700 10 monclient(hunting): picked mon.noname-b con 0x7ff8f4096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:01.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8fa6da700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff8f40a2390 0x7ff8f4096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:01.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8fa6da700 10 monclient(hunting): picked mon.noname-c con 0x7ff8f40a2390 addr v2:172.21.15.50:3300/0 2023-09-30T01:55:01.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8fa6da700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff8f4097360 addr v1:172.21.15.50:6789/0 2023-09-30T01:55:01.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8fa6da700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f40980e0 con 0x7ff8f4097360 2023-09-30T01:55:01.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8fa6da700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:01.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8fa6da700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f40991a0 con 0x7ff8f40a2390 2023-09-30T01:55:01.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8fa6da700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:01.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8fa6da700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f4099480 con 0x7ff8f4096a50 2023-09-30T01:55:01.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8fa6da700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:01.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8fa6da700 10 monclient(hunting): authenticate will time out at 1190.480317s 2023-09-30T01:55:01.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f3fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8f4096a50 0x7ff8f41327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:01.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f3fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8f4096a50 0x7ff8f41327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33408/0 (socket says 172.21.15.7:33408) 2023-09-30T01:55:01.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f3fff700 1 -- 172.21.15.7:0/1194902310 learned_addr learned my addr 172.21.15.7:0/1194902310 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:01.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f3fff700 10 monclient(hunting): get_auth_request con 0x7ff8f4096a50 auth_method 0 2023-09-30T01:55:01.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:01.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:01.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:01.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:01.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:01.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:01.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f37fe700 1 --2- 172.21.15.7:0/1194902310 >> v2:172.21.15.50:3300/0 conn(0x7ff8f40a2390 0x7ff8f4096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:01.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f37fe700 10 monclient(hunting): get_auth_request con 0x7ff8f40a2390 auth_method 0 2023-09-30T01:55:01.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:01.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:01.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:01.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f3fff700 10 monclient(hunting): handle_auth_done global_id 4319 payload 274 2023-09-30T01:55:01.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f3fff700 1 -- 172.21.15.7:0/1194902310 >> v2:172.21.15.50:3300/0 conn(0x7ff8f40a2390 msgr2=0x7ff8f4096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:01.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f3fff700 1 --2- 172.21.15.7:0/1194902310 >> v2:172.21.15.50:3300/0 conn(0x7ff8f40a2390 0x7ff8f4096e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:01.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f3fff700 1 -- 172.21.15.7:0/1194902310 >> v1:172.21.15.50:6789/0 conn(0x7ff8f4097360 legacy=0x7ff8f412ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:01.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f3fff700 10 monclient: _finish_hunting 0 2023-09-30T01:55:01.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f3fff700 1 monclient: found mon.noname-b 2023-09-30T01:55:01.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:01.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:55:01.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.415+0000 7ff8f3fff700 1 -- 172.21.15.7:0/1194902310 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f4097bf0 con 0x7ff8f4096a50 2023-09-30T01:55:01.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.417+0000 7ff8f3fff700 1 -- 172.21.15.7:0/1194902310 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8f4096a50 msgr2=0x7ff8f41327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:01.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8f3fff700 1 --2- 172.21.15.7:0/1194902310 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8f4096a50 0x7ff8f41327a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff8e40099b0 tx=0x7ff8e4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=675ae854e52c59d5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:01.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8f2ffd700 1 -- 172.21.15.7:0/1194902310 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e400ba30 con 0x7ff8f4096a50 2023-09-30T01:55:01.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:01.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8f2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:55:01.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8f2ffd700 10 monclient: dump: 2023-09-30T01:55:01.432 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:01.432 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:01.432 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:01.432 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:01.432 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:01.432 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:01.433 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:01.433 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:01.433 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:01.433 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:01.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8f2ffd700 10 monclient: _finish_auth 0 2023-09-30T01:55:01.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8f2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:01.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8f2ffd700 1 -- 172.21.15.7:0/1194902310 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e400bb90 con 0x7ff8f4096a50 2023-09-30T01:55:01.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8fa6da700 5 monclient: authenticate success, global_id 4319 2023-09-30T01:55:01.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8fa6da700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:55:01.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8f2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:01.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8fa6da700 10 monclient: get_monmap_and_config success 2023-09-30T01:55:01.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8f2ffd700 1 -- 172.21.15.7:0/1194902310 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4011420 con 0x7ff8f4096a50 2023-09-30T01:55:01.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8fa6da700 4 set_mon_vals no callback set 2023-09-30T01:55:01.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:01.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8f2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:55:01.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8f2ffd700 10 monclient: dump: 2023-09-30T01:55:01.434 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:01.434 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:01.435 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:01.435 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:01.435 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:01.435 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:01.435 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:01.435 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:01.435 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:01.435 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:01.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8fa6da700 10 monclient: shutdown 2023-09-30T01:55:01.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8fa6da700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:01.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8fa6da700 1 -- 172.21.15.7:0/1194902310 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8f4096a50 msgr2=0x7ff8f41327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:01.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8fa6da700 1 --2- 172.21.15.7:0/1194902310 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8f4096a50 0x7ff8f41327a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff8e40099b0 tx=0x7ff8e4009250 comp rx=0 tx=0).stop 2023-09-30T01:55:01.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8fa6da700 1 -- 172.21.15.7:0/1194902310 shutdown_connections 2023-09-30T01:55:01.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8fa6da700 1 --2- 172.21.15.7:0/1194902310 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8f4096a50 0x7ff8f41327a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:01.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8fa6da700 1 --2- 172.21.15.7:0/1194902310 >> v2:172.21.15.50:3300/0 conn(0x7ff8f40a2390 0x7ff8f4096e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:01.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8fa6da700 1 -- 172.21.15.7:0/1194902310 >> 172.21.15.7:0/1194902310 conn(0x7ff8f4070cf0 msgr2=0x7ff8f40710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:01.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8fa6da700 1 -- 172.21.15.7:0/1194902310 shutdown_connections 2023-09-30T01:55:01.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.418+0000 7ff8fa6da700 1 -- 172.21.15.7:0/1194902310 wait complete. 2023-09-30T01:55:01.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.419+0000 7ff8fa6da700 10 monclient: build_initial_monmap 2023-09-30T01:55:01.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.419+0000 7ff8fa6da700 10 monclient: monmap: 2023-09-30T01:55:01.436 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:01.437 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:55:01.437 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:55:01.437 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:55:01.437 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:01.437 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:01.437 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:55:01.437 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:55:01.437 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:55:01.437 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:01.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.419+0000 7ff8fa6da700 1 Processor -- start 2023-09-30T01:55:01.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.419+0000 7ff8fa6da700 1 -- start start 2023-09-30T01:55:01.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.419+0000 7ff8fa6da700 10 monclient: init 2023-09-30T01:55:01.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8fa6da700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:01.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8fa6da700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:55:01.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8fa6da700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8f4096a50 0x7ff8f412e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:01.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8fa6da700 10 monclient(hunting): picked mon.noname-c con 0x7ff8f4096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:01.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8fa6da700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8f4097360 0x7ff8f409dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:01.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8fa6da700 10 monclient(hunting): picked mon.noname-b con 0x7ff8f4097360 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:55:01.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8fa6da700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8f40a2390 0x7ff8f409e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:01.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8fa6da700 10 monclient(hunting): picked mon.noname-a con 0x7ff8f40a2390 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:55:01.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8fa6da700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:01.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8fa6da700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f41334e0 con 0x7ff8f40a2390 2023-09-30T01:55:01.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8fa6da700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:01.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8fa6da700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f412f030 con 0x7ff8f4097360 2023-09-30T01:55:01.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8fa6da700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:01.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8fa6da700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f40a1670 con 0x7ff8f4096a50 2023-09-30T01:55:01.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8fa6da700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:01.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8fa6da700 10 monclient(hunting): authenticate will time out at 1190.485378s 2023-09-30T01:55:01.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f3fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8f4096a50 0x7ff8f412e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:01.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f8c77700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8f40a2390 0x7ff8f409e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:01.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f3fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8f4096a50 0x7ff8f412e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:33420/0 (socket says 172.21.15.7:33420) 2023-09-30T01:55:01.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f3fff700 1 -- 172.21.15.7:0/3018304471 learned_addr learned my addr 172.21.15.7:0/3018304471 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:01.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f3fff700 10 monclient(hunting): get_auth_request con 0x7ff8f4096a50 auth_method 0 2023-09-30T01:55:01.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:01.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:01.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:01.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f8c77700 10 monclient(hunting): get_auth_request con 0x7ff8f40a2390 auth_method 0 2023-09-30T01:55:01.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f8c77700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:01.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f8c77700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:01.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f8c77700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:01.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:01.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:01.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:01.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f8c77700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:01.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f8c77700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:01.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f8c77700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:01.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f37fe700 1 --2- 172.21.15.7:0/3018304471 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8f4097360 0x7ff8f409dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:01.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f37fe700 10 monclient(hunting): get_auth_request con 0x7ff8f4097360 auth_method 0 2023-09-30T01:55:01.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:01.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:01.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:01.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f3fff700 10 monclient(hunting): handle_auth_done global_id 4322 payload 824 2023-09-30T01:55:01.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f3fff700 1 -- 172.21.15.7:0/3018304471 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8f4097360 msgr2=0x7ff8f409dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:01.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f3fff700 1 --2- 172.21.15.7:0/3018304471 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8f4097360 0x7ff8f409dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:01.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f3fff700 1 -- 172.21.15.7:0/3018304471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8f40a2390 msgr2=0x7ff8f409e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:01.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f3fff700 1 --2- 172.21.15.7:0/3018304471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8f40a2390 0x7ff8f409e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:01.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f3fff700 10 monclient: _finish_hunting 0 2023-09-30T01:55:01.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f3fff700 1 monclient: found mon.noname-c 2023-09-30T01:55:01.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:01.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f3fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:55:01.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f3fff700 1 -- 172.21.15.7:0/3018304471 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e40094f0 con 0x7ff8f4096a50 2023-09-30T01:55:01.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.420+0000 7ff8f8c77700 1 --2- 172.21.15.7:0/3018304471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8f40a2390 0x7ff8f409e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:55:01.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8f3fff700 1 -- 172.21.15.7:0/3018304471 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8f4096a50 msgr2=0x7ff8f412e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:01.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8f3fff700 1 --2- 172.21.15.7:0/3018304471 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8f4096a50 0x7ff8f412e030 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4099810 tx=0x7ff8e4016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:01.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8ebfff700 1 -- 172.21.15.7:0/3018304471 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4011420 con 0x7ff8f4096a50 2023-09-30T01:55:01.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8ebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:01.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8ebfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:55:01.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8ebfff700 10 monclient: dump: 2023-09-30T01:55:01.444 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:01.444 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:01.445 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:01.445 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:01.445 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:01.445 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:01.445 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:01.446 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:01.446 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:01.446 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:01.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8ebfff700 10 monclient: _finish_auth 0 2023-09-30T01:55:01.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8ebfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:01.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8fa6da700 5 monclient: authenticate success, global_id 4322 2023-09-30T01:55:01.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8ebfff700 1 -- 172.21.15.7:0/3018304471 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e4020070 con 0x7ff8f4096a50 2023-09-30T01:55:01.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8ebfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:01.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8ebfff700 1 -- 172.21.15.7:0/3018304471 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e401b450 con 0x7ff8f4096a50 2023-09-30T01:55:01.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8fa6da700 10 monclient: _renew_subs 2023-09-30T01:55:01.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8fa6da700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:01.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8fa6da700 1 -- 172.21.15.7:0/3018304471 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f40a1390 con 0x7ff8f4096a50 2023-09-30T01:55:01.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8f27fc700 4 set_mon_vals no callback set 2023-09-30T01:55:01.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8ebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:01.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8ebfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:55:01.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8ebfff700 10 monclient: dump: 2023-09-30T01:55:01.448 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:01.448 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:01.448 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:01.448 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:01.448 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:01.448 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:01.448 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:01.448 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:01.449 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:01.449 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:01.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8fa6da700 10 monclient: _renew_subs 2023-09-30T01:55:01.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8fa6da700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:01.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.423+0000 7ff8fa6da700 1 -- 172.21.15.7:0/3018304471 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f4136e70 con 0x7ff8f4096a50 2023-09-30T01:55:01.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.424+0000 7ff8ebfff700 1 -- 172.21.15.7:0/3018304471 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7ff8e4022030 con 0x7ff8f4096a50 2023-09-30T01:55:01.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.424+0000 7ff8ebfff700 1 --2- 172.21.15.7:0/3018304471 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff8d40357e0 0x7ff8d4037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:01.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.424+0000 7ff8fa6da700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:01.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.424+0000 7ff8fa6da700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:01.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.424+0000 7ff8fa6da700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:01.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.424+0000 7ff8fa6da700 1 -- 172.21.15.7:0/3018304471 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f40a4930 con 0x7ff8f4096a50 2023-09-30T01:55:01.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.424+0000 7ff8ebfff700 1 -- 172.21.15.7:0/3018304471 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7ff8e4049970 con 0x7ff8f4096a50 2023-09-30T01:55:01.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.424+0000 7ff8f37fe700 1 --2- 172.21.15.7:0/3018304471 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff8d40357e0 0x7ff8d4037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:01.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.424+0000 7ff8f37fe700 10 monclient: get_auth_request con 0x7ff8d40357e0 auth_method 0 2023-09-30T01:55:01.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.427+0000 7ff8f37fe700 1 --2- 172.21.15.7:0/3018304471 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff8d40357e0 0x7ff8d4037c90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff8f409f4f0 tx=0x7ff8dc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:01.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.429+0000 7ff8ebfff700 1 -- 172.21.15.7:0/3018304471 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff8e401a9c0 con 0x7ff8f4096a50 2023-09-30T01:55:01.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.429+0000 7ff8ebfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:01.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.429+0000 7ff8ebfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:55:01.662 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.662+0000 7ff8fa6da700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:55:01.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.662+0000 7ff8fa6da700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:01.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.662+0000 7ff8fa6da700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:01.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.662+0000 7ff8fa6da700 1 -- 172.21.15.7:0/3018304471 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff8f40a4930 con 0x7ff8f4096a50 2023-09-30T01:55:01.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.662+0000 7ff8ebfff700 1 -- 172.21.15.7:0/3018304471 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff8e40193f0 con 0x7ff8f4096a50 2023-09-30T01:55:01.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.662+0000 7ff8ebfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:01.663 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.662+0000 7ff8ebfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:55:01.664 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:55:01.664 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:55:01.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.665+0000 7ff8fa6da700 1 -- 172.21.15.7:0/3018304471 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff8d40357e0 msgr2=0x7ff8d4037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:01.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.665+0000 7ff8fa6da700 1 --2- 172.21.15.7:0/3018304471 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff8d40357e0 0x7ff8d4037c90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff8f409f4f0 tx=0x7ff8dc008040 comp rx=0 tx=0).stop 2023-09-30T01:55:01.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.665+0000 7ff8fa6da700 10 monclient: shutdown 2023-09-30T01:55:01.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.665+0000 7ff8fa6da700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:01.665 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.665+0000 7ff8fa6da700 1 -- 172.21.15.7:0/3018304471 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8f4096a50 msgr2=0x7ff8f412e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:01.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.665+0000 7ff8fa6da700 1 --2- 172.21.15.7:0/3018304471 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8f4096a50 0x7ff8f412e030 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4099810 tx=0x7ff8e4016000 comp rx=0 tx=0).stop 2023-09-30T01:55:01.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.665+0000 7ff8fa6da700 1 -- 172.21.15.7:0/3018304471 shutdown_connections 2023-09-30T01:55:01.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.665+0000 7ff8fa6da700 1 --2- 172.21.15.7:0/3018304471 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff8d40357e0 0x7ff8d4037c90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:01.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.665+0000 7ff8fa6da700 1 --2- 172.21.15.7:0/3018304471 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff8f40a2390 0x7ff8f409e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:01.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.665+0000 7ff8fa6da700 1 --2- 172.21.15.7:0/3018304471 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff8f4096a50 0x7ff8f412e030 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:01.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.665+0000 7ff8fa6da700 1 --2- 172.21.15.7:0/3018304471 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff8f4097360 0x7ff8f409dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:01.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.665+0000 7ff8fa6da700 1 -- 172.21.15.7:0/3018304471 >> 172.21.15.7:0/3018304471 conn(0x7ff8f4070cf0 msgr2=0x7ff8f4131080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:01.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.665+0000 7ff8fa6da700 1 -- 172.21.15.7:0/3018304471 shutdown_connections 2023-09-30T01:55:01.666 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:01.665+0000 7ff8fa6da700 1 -- 172.21.15.7:0/3018304471 wait complete. 2023-09-30T01:55:04.678 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:55:04.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.809+0000 7f760989a700 10 monclient: get_monmap_and_config 2023-09-30T01:55:04.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.809+0000 7f760989a700 10 monclient: build_initial_monmap 2023-09-30T01:55:04.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.809+0000 7f760989a700 10 monclient: monmap: 2023-09-30T01:55:04.810 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:04.810 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:04.810 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:55:04.810527+0000 2023-09-30T01:55:04.810 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:55:04.810527+0000 2023-09-30T01:55:04.810 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:04.810 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:04.810 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:55:04.811 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:55:04.811 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:55:04.811 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:55:04.811 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:55:04.811 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:04.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f760989a700 1 Processor -- start 2023-09-30T01:55:04.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f760989a700 1 -- start start 2023-09-30T01:55:04.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f760989a700 10 monclient: init 2023-09-30T01:55:04.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f760989a700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:04.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f760989a700 10 monclient: _add_conns ranks=[2,0,3,4,1] 2023-09-30T01:55:04.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f760989a700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7604096a50 0x7f76041327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:04.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f760989a700 10 monclient(hunting): picked mon.noname-b con 0x7f7604096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:04.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f760989a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f76040a2390 addr v1:172.21.15.7:6789/0 2023-09-30T01:55:04.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f760989a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7604097530 0x7f7604139720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:04.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f760989a700 10 monclient(hunting): picked mon.noname-a con 0x7f7604097530 addr v2:172.21.15.7:3300/0 2023-09-30T01:55:04.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f760989a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76040980e0 con 0x7f76040a2390 2023-09-30T01:55:04.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f760989a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:04.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f760989a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76040991a0 con 0x7f7604097530 2023-09-30T01:55:04.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f760989a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:04.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f760989a700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7604099480 con 0x7f7604096a50 2023-09-30T01:55:04.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f760989a700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:04.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f760989a700 10 monclient(hunting): authenticate will time out at 1193.875684s 2023-09-30T01:55:04.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f7602ffd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7604096a50 0x7f76041327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:04.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f76037fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7604097530 0x7f7604139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:04.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f76037fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7604097530 0x7f7604139720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34502/0 (socket says 172.21.15.7:34502) 2023-09-30T01:55:04.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f76037fe700 1 -- 172.21.15.7:0/1798722677 learned_addr learned my addr 172.21.15.7:0/1798722677 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:04.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f76037fe700 10 monclient(hunting): get_auth_request con 0x7f7604097530 auth_method 0 2023-09-30T01:55:04.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f76037fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:04.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f76037fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:04.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f76037fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:04.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f7602ffd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7604096a50 0x7f76041327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:40936/0 (socket says 172.21.15.7:40936) 2023-09-30T01:55:04.819 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f7602ffd700 10 monclient(hunting): get_auth_request con 0x7f7604096a50 auth_method 0 2023-09-30T01:55:04.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f7602ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:04.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f7602ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:04.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f7602ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:04.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f76037fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:04.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f76037fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:04.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f76037fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:04.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f7602ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:04.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f7602ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:04.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.810+0000 7f7602ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:04.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.811+0000 7f7602ffd700 10 monclient(hunting): handle_auth_done global_id 4325 payload 274 2023-09-30T01:55:04.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.811+0000 7f7602ffd700 1 -- 172.21.15.7:0/1798722677 >> v2:172.21.15.7:3300/0 conn(0x7f7604097530 msgr2=0x7f7604139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:04.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.811+0000 7f7602ffd700 1 --2- 172.21.15.7:0/1798722677 >> v2:172.21.15.7:3300/0 conn(0x7f7604097530 0x7f7604139720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:04.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.811+0000 7f7602ffd700 1 -- 172.21.15.7:0/1798722677 >> v1:172.21.15.7:6789/0 conn(0x7f76040a2390 legacy=0x7f7604096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:04.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.811+0000 7f7602ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:55:04.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.811+0000 7f7602ffd700 1 monclient: found mon.noname-b 2023-09-30T01:55:04.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.811+0000 7f7602ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:04.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.811+0000 7f7602ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:55:04.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.811+0000 7f7602ffd700 1 -- 172.21.15.7:0/1798722677 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7604097bf0 con 0x7f7604096a50 2023-09-30T01:55:04.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.811+0000 7f76037fe700 1 --2- 172.21.15.7:0/1798722677 >> v2:172.21.15.7:3300/0 conn(0x7f7604097530 0x7f7604139720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:55:04.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.813+0000 7f7602ffd700 1 -- 172.21.15.7:0/1798722677 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7604096a50 msgr2=0x7f76041327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:04.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.813+0000 7f7602ffd700 1 --2- 172.21.15.7:0/1798722677 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7604096a50 0x7f76041327a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f75f40098e0 tx=0x7f75f40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38eaf508af52412 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:04.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.813+0000 7f7601ffb700 1 -- 172.21.15.7:0/1798722677 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f4011840 con 0x7f7604096a50 2023-09-30T01:55:04.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.813+0000 7f7601ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:04.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.813+0000 7f7601ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:55:04.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.813+0000 7f7601ffb700 10 monclient: dump: 2023-09-30T01:55:04.822 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:04.822 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:04.822 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:04.822 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:04.822 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:04.822 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:04.822 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:04.822 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:04.822 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:04.822 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:04.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.813+0000 7f7601ffb700 10 monclient: _finish_auth 0 2023-09-30T01:55:04.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.813+0000 7f7601ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:04.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.813+0000 7f7601ffb700 1 -- 172.21.15.7:0/1798722677 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f40119a0 con 0x7f7604096a50 2023-09-30T01:55:04.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.813+0000 7f760989a700 5 monclient: authenticate success, global_id 4325 2023-09-30T01:55:04.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.813+0000 7f760989a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:55:04.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.813+0000 7f7601ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:04.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.813+0000 7f760989a700 10 monclient: get_monmap_and_config success 2023-09-30T01:55:04.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.813+0000 7f7601ffb700 1 -- 172.21.15.7:0/1798722677 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f4011d40 con 0x7f7604096a50 2023-09-30T01:55:04.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.813+0000 7f7601ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:04.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.813+0000 7f760989a700 4 set_mon_vals no callback set 2023-09-30T01:55:04.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.813+0000 7f7601ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:55:04.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.813+0000 7f7601ffb700 10 monclient: dump: 2023-09-30T01:55:04.823 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:04.824 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:04.824 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:04.824 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:04.824 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:04.824 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:04.824 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:04.824 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:04.824 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:04.824 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:04.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.814+0000 7f760989a700 10 monclient: shutdown 2023-09-30T01:55:04.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.814+0000 7f760989a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:04.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.814+0000 7f760989a700 1 -- 172.21.15.7:0/1798722677 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7604096a50 msgr2=0x7f76041327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:04.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.814+0000 7f760989a700 1 --2- 172.21.15.7:0/1798722677 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7604096a50 0x7f76041327a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f75f40098e0 tx=0x7f75f40091a0 comp rx=0 tx=0).stop 2023-09-30T01:55:04.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.814+0000 7f760989a700 1 -- 172.21.15.7:0/1798722677 shutdown_connections 2023-09-30T01:55:04.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.814+0000 7f760989a700 1 --2- 172.21.15.7:0/1798722677 >> v2:172.21.15.7:3300/0 conn(0x7f7604097530 0x7f7604139720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:04.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.814+0000 7f760989a700 1 --2- 172.21.15.7:0/1798722677 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7604096a50 0x7f76041327a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:04.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.814+0000 7f760989a700 1 -- 172.21.15.7:0/1798722677 >> 172.21.15.7:0/1798722677 conn(0x7f7604070cf0 msgr2=0x7f76040710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:04.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.814+0000 7f760989a700 1 -- 172.21.15.7:0/1798722677 shutdown_connections 2023-09-30T01:55:04.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.814+0000 7f760989a700 1 -- 172.21.15.7:0/1798722677 wait complete. 2023-09-30T01:55:04.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f760989a700 10 monclient: build_initial_monmap 2023-09-30T01:55:04.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f760989a700 10 monclient: monmap: 2023-09-30T01:55:04.825 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:04.825 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:55:04.825 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:55:04.826 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:55:04.826 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:04.826 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:04.826 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:55:04.826 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:55:04.826 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:55:04.826 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:04.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f760989a700 1 Processor -- start 2023-09-30T01:55:04.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f760989a700 1 -- start start 2023-09-30T01:55:04.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f760989a700 10 monclient: init 2023-09-30T01:55:04.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f760989a700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:04.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f760989a700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:55:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f760989a700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7604096a50 0x7f760409bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f760989a700 10 monclient(hunting): picked mon.noname-c con 0x7f7604096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f760989a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7604097530 0x7f760409c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f760989a700 10 monclient(hunting): picked mon.noname-a con 0x7f7604097530 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:55:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f760989a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76040a2390 0x7f760409c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f760989a700 10 monclient(hunting): picked mon.noname-b con 0x7f76040a2390 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:55:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f760989a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f760989a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76041334e0 con 0x7f7604097530 2023-09-30T01:55:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f760989a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f760989a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7604130120 con 0x7f76040a2390 2023-09-30T01:55:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f760989a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f760989a700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f760409ebb0 con 0x7f7604096a50 2023-09-30T01:55:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f760989a700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f760989a700 10 monclient(hunting): authenticate will time out at 1193.880853s 2023-09-30T01:55:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f7602ffd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7604096a50 0x7f760409bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f76027fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7604097530 0x7f760409c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f7602ffd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7604096a50 0x7f760409bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:40938/0 (socket says 172.21.15.7:40938) 2023-09-30T01:55:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f7602ffd700 1 -- 172.21.15.7:0/1646134080 learned_addr learned my addr 172.21.15.7:0/1646134080 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f7602ffd700 10 monclient(hunting): get_auth_request con 0x7f7604096a50 auth_method 0 2023-09-30T01:55:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f7602ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f7602ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f7602ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f76027fc700 10 monclient(hunting): get_auth_request con 0x7f7604097530 auth_method 0 2023-09-30T01:55:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f76027fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:04.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f76027fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:04.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.815+0000 7f76027fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:04.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f76027fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:04.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f76027fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:04.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f76027fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:04.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f7602ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:04.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f7602ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:04.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f7602ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:04.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f76037fe700 1 --2- 172.21.15.7:0/1646134080 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76040a2390 0x7f760409c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:04.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f76037fe700 10 monclient(hunting): get_auth_request con 0x7f76040a2390 auth_method 0 2023-09-30T01:55:04.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f76037fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:04.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f76037fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:04.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f76037fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:04.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f7602ffd700 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2023-09-30T01:55:04.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f7602ffd700 1 -- 172.21.15.7:0/1646134080 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76040a2390 msgr2=0x7f760409c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:04.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f7602ffd700 1 --2- 172.21.15.7:0/1646134080 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76040a2390 0x7f760409c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:04.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f7602ffd700 1 -- 172.21.15.7:0/1646134080 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7604097530 msgr2=0x7f760409c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:04.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f7602ffd700 1 --2- 172.21.15.7:0/1646134080 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7604097530 0x7f760409c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:04.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f7602ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:55:04.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f7602ffd700 1 monclient: found mon.noname-c 2023-09-30T01:55:04.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f7602ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:04.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f7602ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:55:04.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f7602ffd700 1 -- 172.21.15.7:0/1646134080 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75fc009ae0 con 0x7f7604096a50 2023-09-30T01:55:04.830 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.816+0000 7f76027fc700 1 --2- 172.21.15.7:0/1646134080 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7604097530 0x7f760409c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:55:04.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f7602ffd700 1 -- 172.21.15.7:0/1646134080 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7604096a50 msgr2=0x7f760409bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:04.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f7602ffd700 1 --2- 172.21.15.7:0/1646134080 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7604096a50 0x7f760409bb30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f75f40093f0 tx=0x7f75f4016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:04.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f75eaffd700 1 -- 172.21.15.7:0/1646134080 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f4006af0 con 0x7f7604096a50 2023-09-30T01:55:04.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f75eaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:04.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f75eaffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:55:04.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f75eaffd700 10 monclient: dump: 2023-09-30T01:55:04.833 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:04.833 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:04.833 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:04.833 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:04.833 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:04.833 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:04.833 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:04.833 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:04.833 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:04.834 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f75eaffd700 10 monclient: _finish_auth 0 2023-09-30T01:55:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f75eaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f75eaffd700 1 -- 172.21.15.7:0/1646134080 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f4020070 con 0x7f7604096a50 2023-09-30T01:55:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f760989a700 5 monclient: authenticate success, global_id 4328 2023-09-30T01:55:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f760989a700 10 monclient: _renew_subs 2023-09-30T01:55:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f760989a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f760989a700 1 -- 172.21.15.7:0/1646134080 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f4009420 con 0x7f7604096a50 2023-09-30T01:55:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f75eaffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f75eaffd700 1 -- 172.21.15.7:0/1646134080 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f400bdd0 con 0x7f7604096a50 2023-09-30T01:55:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f75eaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f75eaffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:55:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f75eaffd700 10 monclient: dump: 2023-09-30T01:55:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:04.835 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f76017fa700 4 set_mon_vals no callback set 2023-09-30T01:55:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f760989a700 10 monclient: _renew_subs 2023-09-30T01:55:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f760989a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.818+0000 7f760989a700 1 -- 172.21.15.7:0/1646134080 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f760409dff0 con 0x7f7604096a50 2023-09-30T01:55:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.819+0000 7f75eaffd700 1 -- 172.21.15.7:0/1646134080 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f75f4022030 con 0x7f7604096a50 2023-09-30T01:55:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.819+0000 7f760989a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.819+0000 7f760989a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.819+0000 7f760989a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.819+0000 7f760989a700 1 -- 172.21.15.7:0/1646134080 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76040a61b0 con 0x7f7604096a50 2023-09-30T01:55:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.819+0000 7f75eaffd700 1 --2- 172.21.15.7:0/1646134080 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f75f00357e0 0x7f75f0037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.819+0000 7f75eaffd700 1 -- 172.21.15.7:0/1646134080 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f75f4049940 con 0x7f7604096a50 2023-09-30T01:55:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.820+0000 7f76027fc700 1 --2- 172.21.15.7:0/1646134080 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f75f00357e0 0x7f75f0037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.820+0000 7f76027fc700 10 monclient: get_auth_request con 0x7f75f00357e0 auth_method 0 2023-09-30T01:55:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.822+0000 7f76027fc700 1 --2- 172.21.15.7:0/1646134080 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f75f00357e0 0x7f75f0037c90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f76040a18f0 tx=0x7f75f8007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.824+0000 7f75eaffd700 1 -- 172.21.15.7:0/1646134080 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f75f401a9c0 con 0x7f7604096a50 2023-09-30T01:55:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.824+0000 7f75eaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:04.824+0000 7f75eaffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:55:05.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:05.059+0000 7f760989a700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:55:05.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:05.059+0000 7f760989a700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:05.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:05.059+0000 7f760989a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:05.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:05.059+0000 7f760989a700 1 -- 172.21.15.7:0/1646134080 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f76040a61b0 con 0x7f7604096a50 2023-09-30T01:55:05.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:05.060+0000 7f75eaffd700 1 -- 172.21.15.7:0/1646134080 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f75f40193f0 con 0x7f7604096a50 2023-09-30T01:55:05.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:05.060+0000 7f75eaffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:05.060 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:05.060+0000 7f75eaffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:55:05.061 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:55:05.061 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:55:05.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:05.062+0000 7f760989a700 1 -- 172.21.15.7:0/1646134080 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f75f00357e0 msgr2=0x7f75f0037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:05.062 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:05.062+0000 7f760989a700 1 --2- 172.21.15.7:0/1646134080 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f75f00357e0 0x7f75f0037c90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f76040a18f0 tx=0x7f75f8007ea0 comp rx=0 tx=0).stop 2023-09-30T01:55:05.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:05.062+0000 7f760989a700 10 monclient: shutdown 2023-09-30T01:55:05.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:05.062+0000 7f760989a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:05.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:05.062+0000 7f760989a700 1 -- 172.21.15.7:0/1646134080 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7604096a50 msgr2=0x7f760409bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:05.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:05.062+0000 7f760989a700 1 --2- 172.21.15.7:0/1646134080 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7604096a50 0x7f760409bb30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f75f40093f0 tx=0x7f75f4016000 comp rx=0 tx=0).stop 2023-09-30T01:55:05.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:05.062+0000 7f760989a700 1 -- 172.21.15.7:0/1646134080 shutdown_connections 2023-09-30T01:55:05.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:05.062+0000 7f760989a700 1 --2- 172.21.15.7:0/1646134080 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f75f00357e0 0x7f75f0037c90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:05.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:05.062+0000 7f760989a700 1 --2- 172.21.15.7:0/1646134080 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76040a2390 0x7f760409c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:05.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:05.062+0000 7f760989a700 1 --2- 172.21.15.7:0/1646134080 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7604096a50 0x7f760409bb30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:05.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:05.062+0000 7f760989a700 1 --2- 172.21.15.7:0/1646134080 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7604097530 0x7f760409c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:05.063 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:05.062+0000 7f760989a700 1 -- 172.21.15.7:0/1646134080 >> 172.21.15.7:0/1646134080 conn(0x7f7604070cf0 msgr2=0x7f7604130760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:05.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:05.062+0000 7f760989a700 1 -- 172.21.15.7:0/1646134080 shutdown_connections 2023-09-30T01:55:05.064 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:05.062+0000 7f760989a700 1 -- 172.21.15.7:0/1646134080 wait complete. 2023-09-30T01:55:08.075 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:55:08.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.206+0000 7fae5f94f700 10 monclient: get_monmap_and_config 2023-09-30T01:55:08.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.206+0000 7fae5f94f700 10 monclient: build_initial_monmap 2023-09-30T01:55:08.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.206+0000 7fae5f94f700 10 monclient: monmap: 2023-09-30T01:55:08.206 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:08.206 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:08.206 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:55:08.207071+0000 2023-09-30T01:55:08.206 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:55:08.207071+0000 2023-09-30T01:55:08.206 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:08.206 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:55:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:55:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:55:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:55:08.207 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:55:08.207 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:08.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.206+0000 7fae5f94f700 1 Processor -- start 2023-09-30T01:55:08.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.206+0000 7fae5f94f700 1 -- start start 2023-09-30T01:55:08.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.206+0000 7fae5f94f700 10 monclient: init 2023-09-30T01:55:08.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5f94f700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:08.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5f94f700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-09-30T01:55:08.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5f94f700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fae58096a50 0x7fae58132750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:08.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5f94f700 10 monclient(hunting): picked mon.noname-b con 0x7fae58096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:08.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5f94f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fae58096e20 addr v1:172.21.15.50:6789/0 2023-09-30T01:55:08.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5f94f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fae581376f0 addr v1:172.21.15.7:6789/0 2023-09-30T01:55:08.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5f94f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae58099480 con 0x7fae581376f0 2023-09-30T01:55:08.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5f94f700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae58097bf0 con 0x7fae58096e20 2023-09-30T01:55:08.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5f94f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:08.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5f94f700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae58097920 con 0x7fae58096a50 2023-09-30T01:55:08.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5f94f700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:08.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5f94f700 10 monclient(hunting): authenticate will time out at 1197.272324s 2023-09-30T01:55:08.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5d6eb700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fae58096a50 0x7fae58132750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:08.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5deec700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fae581376f0 0x7fae58139ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37144/0 (socket says 172.21.15.7:37144) 2023-09-30T01:55:08.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5deec700 1 -- 172.21.15.7:0/2956966863 learned_addr learned my addr 172.21.15.7:0/2956966863 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:55:08.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5d6eb700 10 monclient(hunting): get_auth_request con 0x7fae58096a50 auth_method 0 2023-09-30T01:55:08.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5d6eb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:08.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5d6eb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:08.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5d6eb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:08.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5d6eb700 1 -- 172.21.15.7:0/2956966863 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fae58096a50 msgr2=0x7fae58132750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T01:55:08.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5d6eb700 1 -- 172.21.15.7:0/2956966863 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fae58096a50 msgr2=0x7fae58132750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T01:55:08.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5d6eb700 1 --2- 172.21.15.7:0/2956966863 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fae58096a50 0x7fae58132750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:55:08.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.207+0000 7fae5d6eb700 1 --2- 172.21.15.7:0/2956966863 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fae58096a50 0x7fae58132750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:55:08.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.209+0000 7fae5deec700 1 -- 172.21.15.7:0/2956966863 >> v1:172.21.15.7:6789/0 conn(0x7fae581376f0 legacy=0x7fae58139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.209+0000 7fae57fff700 1 -- 172.21.15.7:0/2956966863 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121525553 0 0) 0x7fae58099480 con 0x7fae581376f0 2023-09-30T01:55:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.209+0000 7fae57fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.209+0000 7fae57fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.209+0000 7fae57fff700 10 monclient(hunting): my global_id is 4329 2023-09-30T01:55:08.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.209+0000 7fae57fff700 1 -- 172.21.15.7:0/2956966863 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae40003570 con 0x7fae581376f0 2023-09-30T01:55:08.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.209+0000 7fae5ceea700 1 -- 172.21.15.7:0/2956966863 >> v1:172.21.15.50:6789/0 conn(0x7fae58096e20 legacy=0x7fae58097210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:08.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 1 -- 172.21.15.7:0/2956966863 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3058748169 0 0) 0x7fae40003570 con 0x7fae581376f0 2023-09-30T01:55:08.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 1 -- 172.21.15.7:0/2956966863 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fae58096a50 msgr2=0x7fae58132750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T01:55:08.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 1 --2- 172.21.15.7:0/2956966863 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fae58096a50 0x7fae58132750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:08.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 1 -- 172.21.15.7:0/2956966863 >> v1:172.21.15.50:6789/0 conn(0x7fae58096e20 legacy=0x7fae58097210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:08.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 10 monclient: _finish_hunting 0 2023-09-30T01:55:08.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 1 monclient: found mon.noname-a-legacy 2023-09-30T01:55:08.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:08.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.7:6789/0 2023-09-30T01:55:08.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 1 -- 172.21.15.7:0/2956966863 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae580991a0 con 0x7fae581376f0 2023-09-30T01:55:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 10 monclient: _finish_auth 0 2023-09-30T01:55:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 1 -- 172.21.15.7:0/2956966863 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4135539532 0 0) 0x7fae500017e0 con 0x7fae581376f0 2023-09-30T01:55:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5f94f700 5 monclient: authenticate success, global_id 4329 2023-09-30T01:55:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5f94f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:55:08.225 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:08.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T01:55:08.226 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 10 monclient: dump: 2023-09-30T01:55:08.226 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:08.226 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:08.226 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:08.227 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:08.227 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:08.227 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:08.227 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:08.227 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:08.227 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:08.228 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:08.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] but i'm connected to v1:172.21.15.7:6789/0, reconnecting 2023-09-30T01:55:08.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:08.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 1 -- 172.21.15.7:0/2956966863 >> v1:172.21.15.7:6789/0 conn(0x7fae581376f0 legacy=0x7fae58139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:08.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:55:08.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 1 --2- 172.21.15.7:0/2956966863 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae40003b10 0x7fae40003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:08.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 10 monclient(hunting): picked mon.a con 0x7fae40003b10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:55:08.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 1 --2- 172.21.15.7:0/2956966863 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fae58096a50 0x7fae40004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:08.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 10 monclient(hunting): picked mon.c con 0x7fae58096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:08.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 1 --2- 172.21.15.7:0/2956966863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae40004da0 0x7fae40005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:08.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 10 monclient(hunting): picked mon.b con 0x7fae40004da0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:55:08.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:08.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 1 -- 172.21.15.7:0/2956966863 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae58097920 con 0x7fae40003b10 2023-09-30T01:55:08.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:08.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 1 -- 172.21.15.7:0/2956966863 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4000bcc0 con 0x7fae40004da0 2023-09-30T01:55:08.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:08.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 1 -- 172.21.15.7:0/2956966863 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4000be80 con 0x7fae58096a50 2023-09-30T01:55:08.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:08.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae57fff700 10 monclient(hunting): _finish_auth 0 2023-09-30T01:55:08.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5f94f700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T01:55:08.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5d6eb700 1 --2- 172.21.15.7:0/2956966863 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae40003b10 0x7fae40003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:08.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5d6eb700 10 monclient(hunting): get_auth_request con 0x7fae40003b10 auth_method 0 2023-09-30T01:55:08.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5d6eb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:08.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5d6eb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:08.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5d6eb700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:55:08.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5ceea700 1 --2- 172.21.15.7:0/2956966863 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fae58096a50 0x7fae40004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:08.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5ceea700 10 monclient(hunting): get_auth_request con 0x7fae58096a50 auth_method 0 2023-09-30T01:55:08.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5ceea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:08.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5ceea700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:08.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5ceea700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:55:08.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5d6eb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:08.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5deec700 1 --2- 172.21.15.7:0/2956966863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae40004da0 0x7fae40005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:08.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5d6eb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:08.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5d6eb700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:55:08.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5deec700 10 monclient(hunting): get_auth_request con 0x7fae40004da0 auth_method 0 2023-09-30T01:55:08.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5deec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:08.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5deec700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:08.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5deec700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:55:08.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5ceea700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:08.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5ceea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:08.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5ceea700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:55:08.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5deec700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:08.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5deec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:08.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5deec700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:55:08.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5d6eb700 10 monclient(hunting): handle_auth_done global_id 4329 payload 293 2023-09-30T01:55:08.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5d6eb700 1 -- 172.21.15.7:0/2956966863 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fae58096a50 msgr2=0x7fae40004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:08.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5d6eb700 1 --2- 172.21.15.7:0/2956966863 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fae58096a50 0x7fae40004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:08.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5d6eb700 1 -- 172.21.15.7:0/2956966863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae40004da0 msgr2=0x7fae40005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:08.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5d6eb700 1 --2- 172.21.15.7:0/2956966863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae40004da0 0x7fae40005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:08.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5d6eb700 10 monclient: _finish_hunting 0 2023-09-30T01:55:08.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5d6eb700 1 monclient: found mon.a 2023-09-30T01:55:08.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5d6eb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:08.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5d6eb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:08.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5d6eb700 1 -- 172.21.15.7:0/2956966863 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae40003570 con 0x7fae40003b10 2023-09-30T01:55:08.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5d6eb700 10 monclient: _finish_auth 0 2023-09-30T01:55:08.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.210+0000 7fae5d6eb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:08.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.212+0000 7fae5d6eb700 1 -- 172.21.15.7:0/2956966863 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae40003b10 msgr2=0x7fae40003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:08.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.212+0000 7fae5d6eb700 1 --2- 172.21.15.7:0/2956966863 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae40003b10 0x7fae40003ee0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fae48009590 tx=0x7fae48009720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88caccfe51879347 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:08.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae57fff700 1 -- 172.21.15.7:0/2956966863 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae48009a30 con 0x7fae40003b10 2023-09-30T01:55:08.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae57fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:08.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae57fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:55:08.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae57fff700 10 monclient: dump: 2023-09-30T01:55:08.238 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:08.238 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:08.238 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:08.238 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:08.238 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:08.238 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:08.238 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:08.239 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:08.239 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:08.239 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:08.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae57fff700 1 -- 172.21.15.7:0/2956966863 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae48014400 con 0x7fae40003b10 2023-09-30T01:55:08.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae5f94f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:55:08.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae57fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:08.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae57fff700 1 -- 172.21.15.7:0/2956966863 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae480147a0 con 0x7fae40003b10 2023-09-30T01:55:08.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae5f94f700 10 monclient: get_monmap_and_config success 2023-09-30T01:55:08.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae5f94f700 4 set_mon_vals no callback set 2023-09-30T01:55:08.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae57fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:08.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae57fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:55:08.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae57fff700 10 monclient: dump: 2023-09-30T01:55:08.240 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:08.240 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:08.240 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:08.241 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:08.241 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:08.241 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:08.241 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:08.241 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:08.241 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:08.241 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:08.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae5f94f700 10 monclient: shutdown 2023-09-30T01:55:08.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae5f94f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:08.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae5f94f700 1 -- 172.21.15.7:0/2956966863 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae40003b10 msgr2=0x7fae40003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:08.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae5f94f700 1 --2- 172.21.15.7:0/2956966863 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae40003b10 0x7fae40003ee0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fae48009590 tx=0x7fae48009720 comp rx=0 tx=0).stop 2023-09-30T01:55:08.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae5deec700 1 -- 172.21.15.7:0/2956966863 reap_dead start 2023-09-30T01:55:08.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae5f94f700 1 -- 172.21.15.7:0/2956966863 shutdown_connections 2023-09-30T01:55:08.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae5f94f700 1 -- 172.21.15.7:0/2956966863 >> 172.21.15.7:0/2956966863 conn(0x7fae58070cf0 msgr2=0x7fae580710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:08.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae5f94f700 1 -- 172.21.15.7:0/2956966863 shutdown_connections 2023-09-30T01:55:08.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.213+0000 7fae5f94f700 1 -- 172.21.15.7:0/2956966863 wait complete. 2023-09-30T01:55:08.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.214+0000 7fae5f94f700 10 monclient: build_initial_monmap 2023-09-30T01:55:08.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.214+0000 7fae5f94f700 10 monclient: monmap: 2023-09-30T01:55:08.243 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:08.244 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:55:08.244 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:55:08.244 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:55:08.244 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:08.244 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:08.244 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:55:08.245 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:55:08.245 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:55:08.245 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:08.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.214+0000 7fae5f94f700 1 Processor -- start 2023-09-30T01:55:08.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.214+0000 7fae5f94f700 1 -- start start 2023-09-30T01:55:08.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.214+0000 7fae5f94f700 10 monclient: init 2023-09-30T01:55:08.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5f94f700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:08.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5f94f700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:55:08.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5f94f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae40003b10 0x7fae5809dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:08.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5f94f700 10 monclient(hunting): picked mon.noname-b con 0x7fae40003b10 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:55:08.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5f94f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae5809e200 0x7fae5809e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:08.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5f94f700 10 monclient(hunting): picked mon.noname-a con 0x7fae5809e200 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:55:08.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5f94f700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fae5809eb70 0x7fae58147b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:08.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5f94f700 10 monclient(hunting): picked mon.noname-c con 0x7fae5809eb70 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:08.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5f94f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:08.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5f94f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae580a18a0 con 0x7fae5809e200 2023-09-30T01:55:08.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5f94f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:08.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5f94f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae581334c0 con 0x7fae40003b10 2023-09-30T01:55:08.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5f94f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:08.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5f94f700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae580a5f30 con 0x7fae5809eb70 2023-09-30T01:55:08.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5f94f700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:08.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5f94f700 10 monclient(hunting): authenticate will time out at 1197.280390s 2023-09-30T01:55:08.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5deec700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fae5809eb70 0x7fae58147b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:08.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5ceea700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae5809e200 0x7fae5809e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:08.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5deec700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fae5809eb70 0x7fae58147b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:40962/0 (socket says 172.21.15.7:40962) 2023-09-30T01:55:08.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5deec700 1 -- 172.21.15.7:0/316681952 learned_addr learned my addr 172.21.15.7:0/316681952 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:08.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5deec700 10 monclient(hunting): get_auth_request con 0x7fae5809eb70 auth_method 0 2023-09-30T01:55:08.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5deec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:08.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5deec700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:08.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5deec700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5ceea700 10 monclient(hunting): get_auth_request con 0x7fae5809e200 auth_method 0 2023-09-30T01:55:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5ceea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5ceea700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5ceea700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5deec700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5deec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5deec700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5ceea700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5ceea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5ceea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5d6eb700 1 --2- 172.21.15.7:0/316681952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae40003b10 0x7fae5809dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:08.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5deec700 10 monclient(hunting): handle_auth_done global_id 4331 payload 824 2023-09-30T01:55:08.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5deec700 1 -- 172.21.15.7:0/316681952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae40003b10 msgr2=0x7fae5809dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:08.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5deec700 1 --2- 172.21.15.7:0/316681952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae40003b10 0x7fae5809dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:08.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5deec700 1 -- 172.21.15.7:0/316681952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae5809e200 msgr2=0x7fae5809e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:08.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5deec700 1 --2- 172.21.15.7:0/316681952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae5809e200 0x7fae5809e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:08.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5deec700 10 monclient: _finish_hunting 0 2023-09-30T01:55:08.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5deec700 1 monclient: found mon.noname-c 2023-09-30T01:55:08.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5deec700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:08.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5deec700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:55:08.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5deec700 1 -- 172.21.15.7:0/316681952 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5809fca0 con 0x7fae5809eb70 2023-09-30T01:55:08.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5d6eb700 10 monclient: get_auth_request con 0x7fae40003b10 auth_method 0 2023-09-30T01:55:08.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.215+0000 7fae5d6eb700 1 --2- 172.21.15.7:0/316681952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae40003b10 0x7fae5809dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:55:08.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.217+0000 7fae5deec700 1 -- 172.21.15.7:0/316681952 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fae5809eb70 msgr2=0x7fae58147b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:08.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.218+0000 7fae5deec700 1 --2- 172.21.15.7:0/316681952 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fae5809eb70 0x7fae58147b90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fae50001490 tx=0x7fae58099ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:08.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.218+0000 7fae557fa700 1 -- 172.21.15.7:0/316681952 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae500117a0 con 0x7fae5809eb70 2023-09-30T01:55:08.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.218+0000 7fae557fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:08.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.218+0000 7fae557fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:55:08.250 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.218+0000 7fae557fa700 10 monclient: dump: 2023-09-30T01:55:08.250 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:08.250 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:08.250 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:08.250 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:08.250 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:08.250 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:08.251 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:08.251 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:08.251 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:08.251 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:08.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.218+0000 7fae557fa700 10 monclient: _finish_auth 0 2023-09-30T01:55:08.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.218+0000 7fae557fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:08.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.218+0000 7fae5f94f700 5 monclient: authenticate success, global_id 4331 2023-09-30T01:55:08.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.218+0000 7fae557fa700 1 -- 172.21.15.7:0/316681952 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae50011900 con 0x7fae5809eb70 2023-09-30T01:55:08.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.218+0000 7fae557fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:08.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.218+0000 7fae557fa700 1 -- 172.21.15.7:0/316681952 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae50011cc0 con 0x7fae5809eb70 2023-09-30T01:55:08.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.218+0000 7fae5f94f700 10 monclient: _renew_subs 2023-09-30T01:55:08.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.218+0000 7fae5f94f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:08.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.218+0000 7fae5f94f700 1 -- 172.21.15.7:0/316681952 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae580a0360 con 0x7fae5809eb70 2023-09-30T01:55:08.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.218+0000 7fae557fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:08.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.218+0000 7fae557fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:55:08.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.218+0000 7fae557fa700 10 monclient: dump: 2023-09-30T01:55:08.252 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:08.252 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:08.252 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:08.252 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:08.252 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:08.253 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:08.253 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:08.253 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:08.253 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:08.253 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:08.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.218+0000 7fae577fe700 4 set_mon_vals no callback set 2023-09-30T01:55:08.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.218+0000 7fae5f94f700 10 monclient: _renew_subs 2023-09-30T01:55:08.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.218+0000 7fae5f94f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:08.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.218+0000 7fae5f94f700 1 -- 172.21.15.7:0/316681952 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae5809f720 con 0x7fae5809eb70 2023-09-30T01:55:08.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.219+0000 7fae557fa700 1 -- 172.21.15.7:0/316681952 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fae50013600 con 0x7fae5809eb70 2023-09-30T01:55:08.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.219+0000 7fae5f94f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:08.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.219+0000 7fae5f94f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:08.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.219+0000 7fae5f94f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:08.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.219+0000 7fae557fa700 1 --2- 172.21.15.7:0/316681952 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fae440357e0 0x7fae44037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:08.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.219+0000 7fae5f94f700 1 -- 172.21.15.7:0/316681952 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae580a16e0 con 0x7fae5809eb70 2023-09-30T01:55:08.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.219+0000 7fae557fa700 1 -- 172.21.15.7:0/316681952 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fae50047d20 con 0x7fae5809eb70 2023-09-30T01:55:08.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.219+0000 7fae5d6eb700 1 --2- 172.21.15.7:0/316681952 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fae440357e0 0x7fae44037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:08.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.219+0000 7fae5d6eb700 10 monclient: get_auth_request con 0x7fae440357e0 auth_method 0 2023-09-30T01:55:08.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.222+0000 7fae5d6eb700 1 --2- 172.21.15.7:0/316681952 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fae440357e0 0x7fae44037c90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fae4800afd0 tx=0x7fae48017000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:08.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.224+0000 7fae557fa700 1 -- 172.21.15.7:0/316681952 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fae580a16e0 con 0x7fae5809eb70 2023-09-30T01:55:08.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.224+0000 7fae557fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:08.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.224+0000 7fae557fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:55:08.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.466+0000 7fae5f94f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:55:08.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.466+0000 7fae5f94f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:08.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.466+0000 7fae5f94f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:08.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.466+0000 7fae5f94f700 1 -- 172.21.15.7:0/316681952 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fae580a16e0 con 0x7fae5809eb70 2023-09-30T01:55:08.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.466+0000 7fae557fa700 1 -- 172.21.15.7:0/316681952 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fae580a16e0 con 0x7fae5809eb70 2023-09-30T01:55:08.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.466+0000 7fae557fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:08.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.466+0000 7fae557fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:55:08.467 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:55:08.467 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:55:08.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.469+0000 7fae5f94f700 1 -- 172.21.15.7:0/316681952 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fae440357e0 msgr2=0x7fae44037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:08.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.469+0000 7fae5f94f700 1 --2- 172.21.15.7:0/316681952 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fae440357e0 0x7fae44037c90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fae4800afd0 tx=0x7fae48017000 comp rx=0 tx=0).stop 2023-09-30T01:55:08.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.469+0000 7fae5f94f700 10 monclient: shutdown 2023-09-30T01:55:08.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.469+0000 7fae5f94f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:08.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.469+0000 7fae5f94f700 1 -- 172.21.15.7:0/316681952 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fae5809eb70 msgr2=0x7fae58147b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:08.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.469+0000 7fae5f94f700 1 --2- 172.21.15.7:0/316681952 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fae5809eb70 0x7fae58147b90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fae50001490 tx=0x7fae58099ef0 comp rx=0 tx=0).stop 2023-09-30T01:55:08.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.469+0000 7fae5f94f700 1 -- 172.21.15.7:0/316681952 shutdown_connections 2023-09-30T01:55:08.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.469+0000 7fae5f94f700 1 --2- 172.21.15.7:0/316681952 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fae440357e0 0x7fae44037c90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:08.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.469+0000 7fae5f94f700 1 --2- 172.21.15.7:0/316681952 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fae5809eb70 0x7fae58147b90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:08.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.469+0000 7fae5f94f700 1 --2- 172.21.15.7:0/316681952 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae40003b10 0x7fae5809dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:08.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.469+0000 7fae5f94f700 1 --2- 172.21.15.7:0/316681952 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fae5809e200 0x7fae5809e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:08.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.469+0000 7fae5f94f700 1 -- 172.21.15.7:0/316681952 >> 172.21.15.7:0/316681952 conn(0x7fae58070cf0 msgr2=0x7fae580a3570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:08.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.469+0000 7fae5f94f700 1 -- 172.21.15.7:0/316681952 shutdown_connections 2023-09-30T01:55:08.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:08.469+0000 7fae5f94f700 1 -- 172.21.15.7:0/316681952 wait complete. 2023-09-30T01:55:11.481 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:55:11.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.613+0000 7f3ca1507700 10 monclient: get_monmap_and_config 2023-09-30T01:55:11.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.613+0000 7f3ca1507700 10 monclient: build_initial_monmap 2023-09-30T01:55:11.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.613+0000 7f3ca1507700 10 monclient: monmap: 2023-09-30T01:55:11.614 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:11.614 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:11.614 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:55:11.614138+0000 2023-09-30T01:55:11.614 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:55:11.614138+0000 2023-09-30T01:55:11.614 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:11.614 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:11.614 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:55:11.614 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:55:11.614 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:55:11.614 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:55:11.615 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:55:11.615 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:11.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.613+0000 7f3ca1507700 1 Processor -- start 2023-09-30T01:55:11.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.613+0000 7f3ca1507700 1 -- start start 2023-09-30T01:55:11.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.613+0000 7f3ca1507700 10 monclient: init 2023-09-30T01:55:11.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3ca1507700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:11.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3ca1507700 10 monclient: _add_conns ranks=[4,2,1,0,3] 2023-09-30T01:55:11.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3ca1507700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3c9c09cd10 0x7f3c9c09b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:11.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3ca1507700 10 monclient(hunting): picked mon.noname-c con 0x7f3c9c09cd10 addr v2:172.21.15.50:3300/0 2023-09-30T01:55:11.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3ca1507700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3c9c097050 0x7f3c9c097420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:11.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3ca1507700 10 monclient(hunting): picked mon.noname-b con 0x7f3c9c097050 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:11.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3ca1507700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3c9c097960 addr v1:172.21.15.50:6789/0 2023-09-30T01:55:11.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3ca1507700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c9c09d610 con 0x7f3c9c097960 2023-09-30T01:55:11.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3ca1507700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:11.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3ca1507700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c9c09a310 con 0x7f3c9c09cd10 2023-09-30T01:55:11.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3ca1507700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:11.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3ca1507700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c099ba0 con 0x7f3c9c097050 2023-09-30T01:55:11.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3ca1507700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:11.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3ca1507700 10 monclient(hunting): authenticate will time out at 1200.679326s 2023-09-30T01:55:11.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3c9a7fc700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3c9c097050 0x7f3c9c097420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:11.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3c9a7fc700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3c9c097050 0x7f3c9c097420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:40980/0 (socket says 172.21.15.7:40980) 2023-09-30T01:55:11.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3c9a7fc700 1 -- 172.21.15.7:0/2586789867 learned_addr learned my addr 172.21.15.7:0/2586789867 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:11.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3c9a7fc700 10 monclient(hunting): get_auth_request con 0x7f3c9c097050 auth_method 0 2023-09-30T01:55:11.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3c9a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:11.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3c9a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:11.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3c9a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:11.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3c9a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:11.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3c9a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:11.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3c9a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:11.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3c9affd700 1 --2- 172.21.15.7:0/2586789867 >> v2:172.21.15.50:3300/0 conn(0x7f3c9c09cd10 0x7f3c9c09b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:11.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3c9affd700 10 monclient(hunting): get_auth_request con 0x7f3c9c09cd10 auth_method 0 2023-09-30T01:55:11.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3c9affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:11.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3c9affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:11.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.614+0000 7f3c9affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:11.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.615+0000 7f3c9a7fc700 10 monclient(hunting): handle_auth_done global_id 4334 payload 274 2023-09-30T01:55:11.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.615+0000 7f3c9a7fc700 1 -- 172.21.15.7:0/2586789867 >> v2:172.21.15.50:3300/0 conn(0x7f3c9c09cd10 msgr2=0x7f3c9c09b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:11.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.615+0000 7f3c9a7fc700 1 --2- 172.21.15.7:0/2586789867 >> v2:172.21.15.50:3300/0 conn(0x7f3c9c09cd10 0x7f3c9c09b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:11.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.615+0000 7f3c9a7fc700 1 -- 172.21.15.7:0/2586789867 >> v1:172.21.15.50:6789/0 conn(0x7f3c9c097960 legacy=0x7f3c9c135a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:11.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.615+0000 7f3c9a7fc700 10 monclient: _finish_hunting 0 2023-09-30T01:55:11.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.615+0000 7f3c9a7fc700 1 monclient: found mon.noname-b 2023-09-30T01:55:11.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.615+0000 7f3c9a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:11.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.615+0000 7f3c9a7fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:55:11.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.615+0000 7f3c9a7fc700 1 -- 172.21.15.7:0/2586789867 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9c09d0e0 con 0x7f3c9c097050 2023-09-30T01:55:11.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3c9a7fc700 1 -- 172.21.15.7:0/2586789867 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3c9c097050 msgr2=0x7f3c9c097420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:11.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3c9a7fc700 1 --2- 172.21.15.7:0/2586789867 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3c9c097050 0x7f3c9c097420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000bc50 tx=0x7f3c9000b4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcab174916622677 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:11.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3c99ffb700 1 -- 172.21.15.7:0/2586789867 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9000be60 con 0x7f3c9c097050 2023-09-30T01:55:11.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3c99ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:11.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3c99ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:55:11.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3c99ffb700 10 monclient: dump: 2023-09-30T01:55:11.622 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:11.622 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:11.622 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:11.622 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:11.622 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:11.622 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:11.622 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:11.622 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:11.622 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:11.623 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:11.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3c99ffb700 10 monclient: _finish_auth 0 2023-09-30T01:55:11.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3c99ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:11.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3c99ffb700 1 -- 172.21.15.7:0/2586789867 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c90010470 con 0x7f3c9c097050 2023-09-30T01:55:11.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3ca1507700 5 monclient: authenticate success, global_id 4334 2023-09-30T01:55:11.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3ca1507700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:55:11.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3c99ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:11.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3c99ffb700 1 -- 172.21.15.7:0/2586789867 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c90010830 con 0x7f3c9c097050 2023-09-30T01:55:11.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3ca1507700 10 monclient: get_monmap_and_config success 2023-09-30T01:55:11.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3ca1507700 4 set_mon_vals no callback set 2023-09-30T01:55:11.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3c99ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:11.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3c99ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:55:11.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3c99ffb700 10 monclient: dump: 2023-09-30T01:55:11.624 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:11.624 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:11.624 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:11.624 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:11.624 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:11.624 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:11.624 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:11.624 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:11.624 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:11.624 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:11.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3ca1507700 10 monclient: shutdown 2023-09-30T01:55:11.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3ca1507700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:11.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3ca1507700 1 -- 172.21.15.7:0/2586789867 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3c9c097050 msgr2=0x7f3c9c097420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:11.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.617+0000 7f3ca1507700 1 --2- 172.21.15.7:0/2586789867 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3c9c097050 0x7f3c9c097420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000bc50 tx=0x7f3c9000b4f0 comp rx=0 tx=0).stop 2023-09-30T01:55:11.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.618+0000 7f3ca1507700 1 -- 172.21.15.7:0/2586789867 shutdown_connections 2023-09-30T01:55:11.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.618+0000 7f3ca1507700 1 --2- 172.21.15.7:0/2586789867 >> v2:172.21.15.50:3300/0 conn(0x7f3c9c09cd10 0x7f3c9c09b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:11.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.618+0000 7f3ca1507700 1 --2- 172.21.15.7:0/2586789867 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3c9c097050 0x7f3c9c097420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:11.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.618+0000 7f3ca1507700 1 -- 172.21.15.7:0/2586789867 >> 172.21.15.7:0/2586789867 conn(0x7f3c9c09e0b0 msgr2=0x7f3c9c09e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:11.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.618+0000 7f3ca1507700 1 -- 172.21.15.7:0/2586789867 shutdown_connections 2023-09-30T01:55:11.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.618+0000 7f3ca1507700 1 -- 172.21.15.7:0/2586789867 wait complete. 2023-09-30T01:55:11.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.618+0000 7f3ca1507700 10 monclient: build_initial_monmap 2023-09-30T01:55:11.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.618+0000 7f3ca1507700 10 monclient: monmap: 2023-09-30T01:55:11.630 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:11.630 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:55:11.630 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:55:11.630 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:55:11.630 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:55:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:55:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:55:11.631 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3ca1507700 1 Processor -- start 2023-09-30T01:55:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3ca1507700 1 -- start start 2023-09-30T01:55:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3ca1507700 10 monclient: init 2023-09-30T01:55:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3ca1507700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3ca1507700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:55:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3ca1507700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3c9c097050 0x7f3c9c13e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3ca1507700 10 monclient(hunting): picked mon.noname-c con 0x7f3c9c097050 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:11.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3ca1507700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c9c097960 0x7f3c9c13dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:11.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3ca1507700 10 monclient(hunting): picked mon.noname-b con 0x7f3c9c097960 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:55:11.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3ca1507700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c9c09cd10 0x7f3c9c141a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:11.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3ca1507700 10 monclient(hunting): picked mon.noname-a con 0x7f3c9c09cd10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:55:11.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3ca1507700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:11.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3ca1507700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c122f50 con 0x7f3c9c09cd10 2023-09-30T01:55:11.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3ca1507700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:11.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3ca1507700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c121240 con 0x7f3c9c097960 2023-09-30T01:55:11.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3ca1507700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:11.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3ca1507700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c122c70 con 0x7f3c9c097050 2023-09-30T01:55:11.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3ca1507700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:11.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3ca1507700 10 monclient(hunting): authenticate will time out at 1200.684721s 2023-09-30T01:55:11.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3c9affd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3c9c097050 0x7f3c9c13e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:11.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3c9affd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3c9c097050 0x7f3c9c13e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:40988/0 (socket says 172.21.15.7:40988) 2023-09-30T01:55:11.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3c9affd700 1 -- 172.21.15.7:0/3422495389 learned_addr learned my addr 172.21.15.7:0/3422495389 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:11.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3c9affd700 10 monclient(hunting): get_auth_request con 0x7f3c9c097050 auth_method 0 2023-09-30T01:55:11.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3c9affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:11.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3c9affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:11.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3c9affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:11.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3c9b7fe700 1 --2- 172.21.15.7:0/3422495389 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c9c09cd10 0x7f3c9c141a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:11.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3c9b7fe700 10 monclient(hunting): get_auth_request con 0x7f3c9c09cd10 auth_method 0 2023-09-30T01:55:11.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3c9b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:11.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3c9b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:11.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3c9b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:11.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3c9affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:11.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3c9affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:11.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3c9affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:11.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3c9b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:11.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3c9b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:11.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3c9b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:11.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.619+0000 7f3c9a7fc700 1 --2- 172.21.15.7:0/3422495389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c9c097960 0x7f3c9c13dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:11.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.620+0000 7f3c9a7fc700 10 monclient(hunting): get_auth_request con 0x7f3c9c097960 auth_method 0 2023-09-30T01:55:11.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.620+0000 7f3c9a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:11.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.620+0000 7f3c9a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:11.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.620+0000 7f3c9a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:11.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.620+0000 7f3c9a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:11.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.620+0000 7f3c9a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:11.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.620+0000 7f3c9a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:11.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.620+0000 7f3c9affd700 10 monclient(hunting): handle_auth_done global_id 4337 payload 824 2023-09-30T01:55:11.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.620+0000 7f3c9affd700 1 -- 172.21.15.7:0/3422495389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c9c097960 msgr2=0x7f3c9c13dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:11.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.620+0000 7f3c9affd700 1 --2- 172.21.15.7:0/3422495389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c9c097960 0x7f3c9c13dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:11.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.620+0000 7f3c9affd700 1 -- 172.21.15.7:0/3422495389 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c9c09cd10 msgr2=0x7f3c9c141a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:11.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.620+0000 7f3c9affd700 1 --2- 172.21.15.7:0/3422495389 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c9c09cd10 0x7f3c9c141a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:11.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.620+0000 7f3c9affd700 10 monclient: _finish_hunting 0 2023-09-30T01:55:11.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.620+0000 7f3c9affd700 1 monclient: found mon.noname-c 2023-09-30T01:55:11.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.620+0000 7f3c9affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:11.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.620+0000 7f3c9affd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:55:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.620+0000 7f3c9affd700 1 -- 172.21.15.7:0/3422495389 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9000b790 con 0x7f3c9c097050 2023-09-30T01:55:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.620+0000 7f3c9b7fe700 1 --2- 172.21.15.7:0/3422495389 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c9c09cd10 0x7f3c9c141a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:55:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3c9affd700 1 -- 172.21.15.7:0/3422495389 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3c9c097050 msgr2=0x7f3c9c13e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3c9affd700 1 --2- 172.21.15.7:0/3422495389 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3c9c097050 0x7f3c9c13e520 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00d890 tx=0x7f3c8c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3c82ffd700 1 -- 172.21.15.7:0/3422495389 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c012800 con 0x7f3c9c097050 2023-09-30T01:55:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3c82ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3c82ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:55:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3c82ffd700 10 monclient: dump: 2023-09-30T01:55:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:11.637 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:11.637 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:11.637 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:11.637 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:11.637 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:11.637 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:11.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3c82ffd700 10 monclient: _finish_auth 0 2023-09-30T01:55:11.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3c82ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:11.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3c82ffd700 1 -- 172.21.15.7:0/3422495389 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8c012960 con 0x7f3c9c097050 2023-09-30T01:55:11.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3ca1507700 5 monclient: authenticate success, global_id 4337 2023-09-30T01:55:11.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3ca1507700 10 monclient: _renew_subs 2023-09-30T01:55:11.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3ca1507700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3ca1507700 1 -- 172.21.15.7:0/3422495389 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9c123230 con 0x7f3c9c097050 2023-09-30T01:55:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3c82ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3c82ffd700 1 -- 172.21.15.7:0/3422495389 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c012d00 con 0x7f3c9c097050 2023-09-30T01:55:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3c82ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3c82ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:55:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3c82ffd700 10 monclient: dump: 2023-09-30T01:55:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:11.639 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3c997fa700 4 set_mon_vals no callback set 2023-09-30T01:55:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3ca1507700 10 monclient: _renew_subs 2023-09-30T01:55:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3ca1507700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.622+0000 7f3ca1507700 1 -- 172.21.15.7:0/3422495389 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c9c13e2d0 con 0x7f3c9c097050 2023-09-30T01:55:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.623+0000 7f3c82ffd700 1 -- 172.21.15.7:0/3422495389 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f3c8c014640 con 0x7f3c9c097050 2023-09-30T01:55:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.623+0000 7f3ca1507700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.623+0000 7f3ca1507700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.623+0000 7f3ca1507700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:11.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.623+0000 7f3ca1507700 1 -- 172.21.15.7:0/3422495389 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c9c097460 con 0x7f3c9c097050 2023-09-30T01:55:11.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.623+0000 7f3c82ffd700 1 --2- 172.21.15.7:0/3422495389 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3c880357e0 0x7f3c88037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:11.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.623+0000 7f3c82ffd700 1 -- 172.21.15.7:0/3422495389 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f3c8c048280 con 0x7f3c9c097050 2023-09-30T01:55:11.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.624+0000 7f3c9a7fc700 1 --2- 172.21.15.7:0/3422495389 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3c880357e0 0x7f3c88037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:11.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.624+0000 7f3c9a7fc700 10 monclient: get_auth_request con 0x7f3c880357e0 auth_method 0 2023-09-30T01:55:11.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.626+0000 7f3c9a7fc700 1 --2- 172.21.15.7:0/3422495389 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3c880357e0 0x7f3c88037c90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000bc50 tx=0x7f3c9001a000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:11.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.628+0000 7f3c82ffd700 1 -- 172.21.15.7:0/3422495389 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3c8c012e60 con 0x7f3c9c097050 2023-09-30T01:55:11.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.628+0000 7f3c82ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:11.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.628+0000 7f3c82ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:55:11.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.870+0000 7f3ca1507700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:55:11.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.870+0000 7f3ca1507700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:11.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.870+0000 7f3ca1507700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:11.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.870+0000 7f3ca1507700 1 -- 172.21.15.7:0/3422495389 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3c9c064c10 con 0x7f3c9c097050 2023-09-30T01:55:11.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.871+0000 7f3c82ffd700 1 -- 172.21.15.7:0/3422495389 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3c8c011340 con 0x7f3c9c097050 2023-09-30T01:55:11.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.871+0000 7f3c82ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:11.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.871+0000 7f3c82ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:55:11.872 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:55:11.872 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:55:11.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.873+0000 7f3ca1507700 1 -- 172.21.15.7:0/3422495389 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3c880357e0 msgr2=0x7f3c88037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:11.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.873+0000 7f3ca1507700 1 --2- 172.21.15.7:0/3422495389 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3c880357e0 0x7f3c88037c90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000bc50 tx=0x7f3c9001a000 comp rx=0 tx=0).stop 2023-09-30T01:55:11.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.873+0000 7f3ca1507700 10 monclient: shutdown 2023-09-30T01:55:11.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.873+0000 7f3ca1507700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:11.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.873+0000 7f3ca1507700 1 -- 172.21.15.7:0/3422495389 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3c9c097050 msgr2=0x7f3c9c13e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:11.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.873+0000 7f3ca1507700 1 --2- 172.21.15.7:0/3422495389 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3c9c097050 0x7f3c9c13e520 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00d890 tx=0x7f3c8c008040 comp rx=0 tx=0).stop 2023-09-30T01:55:11.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.873+0000 7f3ca1507700 1 -- 172.21.15.7:0/3422495389 shutdown_connections 2023-09-30T01:55:11.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.873+0000 7f3ca1507700 1 --2- 172.21.15.7:0/3422495389 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3c880357e0 0x7f3c88037c90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:11.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.873+0000 7f3ca1507700 1 --2- 172.21.15.7:0/3422495389 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3c9c09cd10 0x7f3c9c141a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:11.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.873+0000 7f3ca1507700 1 --2- 172.21.15.7:0/3422495389 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3c9c097050 0x7f3c9c13e520 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:11.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.873+0000 7f3ca1507700 1 --2- 172.21.15.7:0/3422495389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3c9c097960 0x7f3c9c13dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:11.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.873+0000 7f3ca1507700 1 -- 172.21.15.7:0/3422495389 >> 172.21.15.7:0/3422495389 conn(0x7f3c9c09e0b0 msgr2=0x7f3c9c06c280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:11.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.873+0000 7f3ca1507700 1 -- 172.21.15.7:0/3422495389 shutdown_connections 2023-09-30T01:55:11.875 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:11.873+0000 7f3ca1507700 1 -- 172.21.15.7:0/3422495389 wait complete. 2023-09-30T01:55:14.886 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:55:15.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.018+0000 7f44f0349700 10 monclient: get_monmap_and_config 2023-09-30T01:55:15.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.018+0000 7f44f0349700 10 monclient: build_initial_monmap 2023-09-30T01:55:15.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.018+0000 7f44f0349700 10 monclient: monmap: 2023-09-30T01:55:15.018 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:15.018 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:15.018 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:55:15.019150+0000 2023-09-30T01:55:15.018 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:55:15.019150+0000 2023-09-30T01:55:15.019 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:15.019 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:15.019 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:55:15.019 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:55:15.019 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:55:15.019 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:55:15.019 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:55:15.019 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:15.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.018+0000 7f44f0349700 1 Processor -- start 2023-09-30T01:55:15.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.018+0000 7f44f0349700 1 -- start start 2023-09-30T01:55:15.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.018+0000 7f44f0349700 10 monclient: init 2023-09-30T01:55:15.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44f0349700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:15.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44f0349700 10 monclient: _add_conns ranks=[4,1,3,0,2] 2023-09-30T01:55:15.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44f0349700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f44e80a29e0 0x7f44e8096a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:15.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44f0349700 10 monclient(hunting): picked mon.noname-c con 0x7f44e80a29e0 addr v2:172.21.15.50:3300/0 2023-09-30T01:55:15.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44f0349700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44e8096f90 addr v1:172.21.15.50:6789/0 2023-09-30T01:55:15.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44f0349700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f44e8097390 0x7f44e8132930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:15.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44f0349700 10 monclient(hunting): picked mon.noname-a con 0x7f44e8097390 addr v2:172.21.15.7:3300/0 2023-09-30T01:55:15.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44f0349700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e8069cc0 con 0x7f44e8096f90 2023-09-30T01:55:15.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44f0349700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:15.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44f0349700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44e8098ea0 con 0x7f44e8097390 2023-09-30T01:55:15.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44f0349700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:15.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44f0349700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44e80a2db0 con 0x7f44e80a29e0 2023-09-30T01:55:15.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44f0349700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:15.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44f0349700 10 monclient(hunting): authenticate will time out at 1204.084539s 2023-09-30T01:55:15.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44ee8e6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f44e8097390 0x7f44e8132930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:15.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44ee8e6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f44e8097390 0x7f44e8132930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37062/0 (socket says 172.21.15.7:37062) 2023-09-30T01:55:15.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44ee8e6700 1 -- 172.21.15.7:0/759100571 learned_addr learned my addr 172.21.15.7:0/759100571 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:15.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44ee8e6700 10 monclient(hunting): get_auth_request con 0x7f44e8097390 auth_method 0 2023-09-30T01:55:15.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44ee8e6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:15.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44ee8e6700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:15.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44ee8e6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:15.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44ee0e5700 1 --2- 172.21.15.7:0/759100571 >> v2:172.21.15.50:3300/0 conn(0x7f44e80a29e0 0x7f44e8096a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:15.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44ee0e5700 10 monclient(hunting): get_auth_request con 0x7f44e80a29e0 auth_method 0 2023-09-30T01:55:15.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44ee0e5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:15.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44ee0e5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:15.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44ee0e5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:15.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44ee8e6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:15.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44ee8e6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:15.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44ee8e6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:15.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44ee0e5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:15.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44ee0e5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:15.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.019+0000 7f44ee0e5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:15.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.020+0000 7f44ee8e6700 10 monclient(hunting): handle_auth_done global_id 4338 payload 274 2023-09-30T01:55:15.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.020+0000 7f44ee8e6700 1 -- 172.21.15.7:0/759100571 >> v2:172.21.15.50:3300/0 conn(0x7f44e80a29e0 msgr2=0x7f44e8096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:15.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.020+0000 7f44ee8e6700 1 --2- 172.21.15.7:0/759100571 >> v2:172.21.15.50:3300/0 conn(0x7f44e80a29e0 0x7f44e8096a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:15.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.020+0000 7f44ee8e6700 1 -- 172.21.15.7:0/759100571 >> v1:172.21.15.50:6789/0 conn(0x7f44e8096f90 legacy=0x7f44e8132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:15.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.020+0000 7f44ee8e6700 10 monclient: _finish_hunting 0 2023-09-30T01:55:15.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.020+0000 7f44ee8e6700 1 monclient: found mon.noname-a 2023-09-30T01:55:15.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.020+0000 7f44ee8e6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:15.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.020+0000 7f44ee8e6700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:55:15.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.020+0000 7f44ee8e6700 1 -- 172.21.15.7:0/759100571 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e8098270 con 0x7f44e8097390 2023-09-30T01:55:15.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44ee8e6700 1 -- 172.21.15.7:0/759100571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e8097390 msgr2=0x7f44e8132930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:15.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44ee8e6700 1 --2- 172.21.15.7:0/759100571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e8097390 0x7f44e8132930 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f44e8071ca0 tx=0x7f44e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5511748685ade03c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:15.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44ed0e3700 1 -- 172.21.15.7:0/759100571 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e0009e70 con 0x7f44e8097390 2023-09-30T01:55:15.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44ed0e3700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:15.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44ed0e3700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:55:15.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44ed0e3700 10 monclient: dump: 2023-09-30T01:55:15.033 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:15.033 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:15.033 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:15.033 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:15.033 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:15.033 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:15.034 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:15.034 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:15.034 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:15.034 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:15.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44ed0e3700 10 monclient: _finish_auth 0 2023-09-30T01:55:15.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44ed0e3700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:15.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44ed0e3700 1 -- 172.21.15.7:0/759100571 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e000ba30 con 0x7f44e8097390 2023-09-30T01:55:15.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44f0349700 5 monclient: authenticate success, global_id 4338 2023-09-30T01:55:15.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44f0349700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:55:15.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44ed0e3700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:15.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44ed0e3700 1 -- 172.21.15.7:0/759100571 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e000bdf0 con 0x7f44e8097390 2023-09-30T01:55:15.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44f0349700 10 monclient: get_monmap_and_config success 2023-09-30T01:55:15.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44f0349700 4 set_mon_vals no callback set 2023-09-30T01:55:15.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44ed0e3700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:15.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44ed0e3700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:55:15.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44ed0e3700 10 monclient: dump: 2023-09-30T01:55:15.035 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:15.035 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:15.035 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:15.035 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:15.035 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:15.035 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:15.035 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:15.036 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:15.036 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:15.036 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:15.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44f0349700 10 monclient: shutdown 2023-09-30T01:55:15.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44f0349700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:15.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44f0349700 1 -- 172.21.15.7:0/759100571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e8097390 msgr2=0x7f44e8132930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:15.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.022+0000 7f44f0349700 1 --2- 172.21.15.7:0/759100571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e8097390 0x7f44e8132930 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f44e8071ca0 tx=0x7f44e0009250 comp rx=0 tx=0).stop 2023-09-30T01:55:15.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.023+0000 7f44f0349700 1 -- 172.21.15.7:0/759100571 shutdown_connections 2023-09-30T01:55:15.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.023+0000 7f44f0349700 1 --2- 172.21.15.7:0/759100571 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e8097390 0x7f44e8132930 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:15.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.023+0000 7f44f0349700 1 --2- 172.21.15.7:0/759100571 >> v2:172.21.15.50:3300/0 conn(0x7f44e80a29e0 0x7f44e8096a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:15.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.023+0000 7f44f0349700 1 -- 172.21.15.7:0/759100571 >> 172.21.15.7:0/759100571 conn(0x7f44e80710d0 msgr2=0x7f44e80714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:15.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.023+0000 7f44f0349700 1 -- 172.21.15.7:0/759100571 shutdown_connections 2023-09-30T01:55:15.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.023+0000 7f44f0349700 1 -- 172.21.15.7:0/759100571 wait complete. 2023-09-30T01:55:15.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.023+0000 7f44f0349700 10 monclient: build_initial_monmap 2023-09-30T01:55:15.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.023+0000 7f44f0349700 10 monclient: monmap: 2023-09-30T01:55:15.037 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:15.037 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:55:15.037 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:55:15.037 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:55:15.037 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:15.037 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:15.037 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:55:15.037 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:55:15.038 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:55:15.038 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:15.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.023+0000 7f44f0349700 1 Processor -- start 2023-09-30T01:55:15.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44f0349700 1 -- start start 2023-09-30T01:55:15.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44f0349700 10 monclient: init 2023-09-30T01:55:15.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44f0349700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:15.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44f0349700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:55:15.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44f0349700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f44e8096f90 0x7f44e80a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:15.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44f0349700 10 monclient(hunting): picked mon.noname-c con 0x7f44e8096f90 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:15.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44f0349700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44e8097390 0x7f44e8141ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:15.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44f0349700 10 monclient(hunting): picked mon.noname-b con 0x7f44e8097390 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:55:15.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44f0349700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e80a29e0 0x7f44e8130cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:15.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44f0349700 10 monclient(hunting): picked mon.noname-a con 0x7f44e80a29e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:55:15.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44f0349700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:15.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44f0349700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80a1250 con 0x7f44e80a29e0 2023-09-30T01:55:15.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44f0349700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:15.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44f0349700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e8142040 con 0x7f44e8097390 2023-09-30T01:55:15.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44f0349700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:15.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44f0349700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e811eea0 con 0x7f44e8096f90 2023-09-30T01:55:15.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44f0349700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:15.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44f0349700 10 monclient(hunting): authenticate will time out at 1204.089440s 2023-09-30T01:55:15.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee8e6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e80a29e0 0x7f44e8130cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:15.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee0e5700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f44e8096f90 0x7f44e80a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:15.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee0e5700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f44e8096f90 0x7f44e80a09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38478/0 (socket says 172.21.15.7:38478) 2023-09-30T01:55:15.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee0e5700 1 -- 172.21.15.7:0/3808097062 learned_addr learned my addr 172.21.15.7:0/3808097062 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:15.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee0e5700 10 monclient(hunting): get_auth_request con 0x7f44e8096f90 auth_method 0 2023-09-30T01:55:15.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee8e6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e80a29e0 0x7f44e8130cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37076/0 (socket says 172.21.15.7:37076) 2023-09-30T01:55:15.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee0e5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:15.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee0e5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:15.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee0e5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:15.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee8e6700 10 monclient(hunting): get_auth_request con 0x7f44e80a29e0 auth_method 0 2023-09-30T01:55:15.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee8e6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:15.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee8e6700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:15.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee8e6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:15.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee8e6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:15.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee8e6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:15.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee8e6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:15.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee0e5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:15.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee0e5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:15.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee0e5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:15.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ed8e4700 1 --2- 172.21.15.7:0/3808097062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44e8097390 0x7f44e8141ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:15.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee8e6700 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2023-09-30T01:55:15.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee8e6700 1 -- 172.21.15.7:0/3808097062 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f44e8096f90 msgr2=0x7f44e80a09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:15.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee8e6700 1 --2- 172.21.15.7:0/3808097062 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f44e8096f90 0x7f44e80a09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:15.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee8e6700 1 -- 172.21.15.7:0/3808097062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44e8097390 msgr2=0x7f44e8141ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:15.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee8e6700 1 --2- 172.21.15.7:0/3808097062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44e8097390 0x7f44e8141ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:15.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee8e6700 10 monclient: _finish_hunting 0 2023-09-30T01:55:15.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee8e6700 1 monclient: found mon.noname-a 2023-09-30T01:55:15.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee8e6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:15.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee8e6700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:55:15.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ee8e6700 1 -- 172.21.15.7:0/3808097062 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e00094f0 con 0x7f44e80a29e0 2023-09-30T01:55:15.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ed8e4700 10 monclient: get_auth_request con 0x7f44e8097390 auth_method 0 2023-09-30T01:55:15.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.024+0000 7f44ed8e4700 1 --2- 172.21.15.7:0/3808097062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44e8097390 0x7f44e8141ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:55:15.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44ee8e6700 1 -- 172.21.15.7:0/3808097062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e80a29e0 msgr2=0x7f44e8130cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:15.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44ee8e6700 1 --2- 172.21.15.7:0/3808097062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e80a29e0 0x7f44e8130cf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f44e0015000 tx=0x7f44e0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:15.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44e67fc700 1 -- 172.21.15.7:0/3808097062 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e0011b90 con 0x7f44e80a29e0 2023-09-30T01:55:15.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44e67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:15.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44e67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:55:15.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44e67fc700 10 monclient: dump: 2023-09-30T01:55:15.043 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:15.043 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:15.043 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:15.043 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:15.043 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:15.043 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:15.043 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:15.043 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:15.043 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:15.043 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:15.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44e67fc700 10 monclient: _finish_auth 0 2023-09-30T01:55:15.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44e67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:15.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44e67fc700 1 -- 172.21.15.7:0/3808097062 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e0020070 con 0x7f44e80a29e0 2023-09-30T01:55:15.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44f0349700 5 monclient: authenticate success, global_id 4341 2023-09-30T01:55:15.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44e67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:15.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44f0349700 10 monclient: _renew_subs 2023-09-30T01:55:15.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44e67fc700 1 -- 172.21.15.7:0/3808097062 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e001b450 con 0x7f44e80a29e0 2023-09-30T01:55:15.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44f0349700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:15.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44f0349700 1 -- 172.21.15.7:0/3808097062 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e811f160 con 0x7f44e80a29e0 2023-09-30T01:55:15.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44e67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:15.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44e67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:55:15.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44e67fc700 10 monclient: dump: 2023-09-30T01:55:15.045 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:15.045 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:15.045 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:15.045 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:15.045 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:15.045 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:15.045 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:15.045 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:15.045 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:15.045 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:15.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44ec8e2700 4 set_mon_vals no callback set 2023-09-30T01:55:15.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44f0349700 10 monclient: _renew_subs 2023-09-30T01:55:15.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44f0349700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:15.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.027+0000 7f44f0349700 1 -- 172.21.15.7:0/3808097062 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e811d2f0 con 0x7f44e80a29e0 2023-09-30T01:55:15.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.028+0000 7f44e67fc700 1 -- 172.21.15.7:0/3808097062 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f44e0022030 con 0x7f44e80a29e0 2023-09-30T01:55:15.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.028+0000 7f44f0349700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:15.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.028+0000 7f44f0349700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:15.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.028+0000 7f44f0349700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:15.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.028+0000 7f44f0349700 1 -- 172.21.15.7:0/3808097062 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e81318e0 con 0x7f44e80a29e0 2023-09-30T01:55:15.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.028+0000 7f44e67fc700 1 --2- 172.21.15.7:0/3808097062 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f44d40357e0 0x7f44d4037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:15.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.028+0000 7f44e67fc700 1 -- 172.21.15.7:0/3808097062 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f44e0049af0 con 0x7f44e80a29e0 2023-09-30T01:55:15.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.028+0000 7f44ee0e5700 1 --2- 172.21.15.7:0/3808097062 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f44d40357e0 0x7f44d4037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:15.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.028+0000 7f44ee0e5700 10 monclient: get_auth_request con 0x7f44d40357e0 auth_method 0 2023-09-30T01:55:15.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.031+0000 7f44ee0e5700 1 --2- 172.21.15.7:0/3808097062 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f44d40357e0 0x7f44d4037c90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f44d80099f0 tx=0x7f44d800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:15.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.033+0000 7f44e67fc700 1 -- 172.21.15.7:0/3808097062 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f44e00249c0 con 0x7f44e80a29e0 2023-09-30T01:55:15.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.033+0000 7f44e67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:15.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.033+0000 7f44e67fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:55:15.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.273+0000 7f44f0349700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:55:15.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.273+0000 7f44f0349700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:15.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.273+0000 7f44f0349700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:15.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.273+0000 7f44f0349700 1 -- 172.21.15.7:0/3808097062 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f44e81318e0 con 0x7f44e80a29e0 2023-09-30T01:55:15.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.273+0000 7f44e67fc700 1 -- 172.21.15.7:0/3808097062 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f44e00193f0 con 0x7f44e80a29e0 2023-09-30T01:55:15.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.273+0000 7f44e67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:15.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.273+0000 7f44e67fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:55:15.274 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:55:15.274 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:55:15.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.276+0000 7f44f0349700 1 -- 172.21.15.7:0/3808097062 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f44d40357e0 msgr2=0x7f44d4037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:15.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.276+0000 7f44f0349700 1 --2- 172.21.15.7:0/3808097062 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f44d40357e0 0x7f44d4037c90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f44d80099f0 tx=0x7f44d800b040 comp rx=0 tx=0).stop 2023-09-30T01:55:15.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.276+0000 7f44f0349700 10 monclient: shutdown 2023-09-30T01:55:15.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.276+0000 7f44f0349700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:15.276 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.276+0000 7f44f0349700 1 -- 172.21.15.7:0/3808097062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e80a29e0 msgr2=0x7f44e8130cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:15.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.276+0000 7f44f0349700 1 --2- 172.21.15.7:0/3808097062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e80a29e0 0x7f44e8130cf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f44e0015000 tx=0x7f44e0016000 comp rx=0 tx=0).stop 2023-09-30T01:55:15.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.276+0000 7f44f0349700 1 -- 172.21.15.7:0/3808097062 shutdown_connections 2023-09-30T01:55:15.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.276+0000 7f44f0349700 1 --2- 172.21.15.7:0/3808097062 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f44d40357e0 0x7f44d4037c90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:15.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.276+0000 7f44f0349700 1 --2- 172.21.15.7:0/3808097062 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f44e80a29e0 0x7f44e8130cf0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:15.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.276+0000 7f44f0349700 1 --2- 172.21.15.7:0/3808097062 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f44e8096f90 0x7f44e80a09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:15.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.276+0000 7f44f0349700 1 --2- 172.21.15.7:0/3808097062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44e8097390 0x7f44e8141ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:15.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.276+0000 7f44f0349700 1 -- 172.21.15.7:0/3808097062 >> 172.21.15.7:0/3808097062 conn(0x7f44e80710d0 msgr2=0x7f44e80a3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:15.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.276+0000 7f44f0349700 1 -- 172.21.15.7:0/3808097062 shutdown_connections 2023-09-30T01:55:15.277 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:15.276+0000 7f44f0349700 1 -- 172.21.15.7:0/3808097062 wait complete. 2023-09-30T01:55:18.288 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:55:18.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.420+0000 7fb42f4a7700 10 monclient: get_monmap_and_config 2023-09-30T01:55:18.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.420+0000 7fb42f4a7700 10 monclient: build_initial_monmap 2023-09-30T01:55:18.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.421+0000 7fb42f4a7700 10 monclient: monmap: 2023-09-30T01:55:18.422 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:18.422 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:18.422 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:55:18.422035+0000 2023-09-30T01:55:18.423 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:55:18.422035+0000 2023-09-30T01:55:18.423 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:18.423 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:18.423 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:55:18.423 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:55:18.423 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:55:18.423 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:55:18.423 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:55:18.423 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:18.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.421+0000 7fb42f4a7700 1 Processor -- start 2023-09-30T01:55:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.421+0000 7fb42f4a7700 1 -- start start 2023-09-30T01:55:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.421+0000 7fb42f4a7700 10 monclient: init 2023-09-30T01:55:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42f4a7700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42f4a7700 10 monclient: _add_conns ranks=[2,3,0,1,4] 2023-09-30T01:55:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42f4a7700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb428096a50 0x7fb428132680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42f4a7700 10 monclient(hunting): picked mon.noname-b con 0x7fb428096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42f4a7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb428096e20 0x7fb4280971f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42f4a7700 10 monclient(hunting): picked mon.noname-a con 0x7fb428096e20 addr v2:172.21.15.7:3300/0 2023-09-30T01:55:18.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42f4a7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb42812ef00 addr v1:172.21.15.7:6789/0 2023-09-30T01:55:18.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42f4a7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb428099440 con 0x7fb42812ef00 2023-09-30T01:55:18.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42f4a7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:18.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42f4a7700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb428097b40 con 0x7fb428096e20 2023-09-30T01:55:18.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42f4a7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:18.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42f4a7700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb428099160 con 0x7fb428096a50 2023-09-30T01:55:18.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42f4a7700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:18.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42f4a7700 10 monclient(hunting): authenticate will time out at 1207.487293s 2023-09-30T01:55:18.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42d243700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb428096a50 0x7fb428132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:18.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42ca42700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb428096e20 0x7fb4280971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:18.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42da44700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb42812ef00 0x7fb42812f370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33814/0 (socket says 172.21.15.7:33814) 2023-09-30T01:55:18.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42da44700 1 -- 172.21.15.7:0/1714484288 learned_addr learned my addr 172.21.15.7:0/1714484288 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:55:18.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42ca42700 10 monclient(hunting): get_auth_request con 0x7fb428096e20 auth_method 0 2023-09-30T01:55:18.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42ca42700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:18.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42ca42700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:18.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42ca42700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:18.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42d243700 10 monclient(hunting): get_auth_request con 0x7fb428096a50 auth_method 0 2023-09-30T01:55:18.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42d243700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:18.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42d243700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:18.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42d243700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:18.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42ca42700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:18.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42ca42700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:18.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42ca42700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:18.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42d243700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:18.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42d243700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:18.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42d243700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:18.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42ca42700 10 monclient(hunting): handle_auth_done global_id 4344 payload 274 2023-09-30T01:55:18.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42ca42700 1 -- 172.21.15.7:0/1714484288 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb428096a50 msgr2=0x7fb428132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:18.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42ca42700 1 --2- 172.21.15.7:0/1714484288 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb428096a50 0x7fb428132680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:18.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42ca42700 1 -- 172.21.15.7:0/1714484288 >> v1:172.21.15.7:6789/0 conn(0x7fb42812ef00 legacy=0x7fb42812f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:18.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42ca42700 10 monclient: _finish_hunting 0 2023-09-30T01:55:18.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42ca42700 1 monclient: found mon.noname-a 2023-09-30T01:55:18.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42ca42700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:18.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42ca42700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:55:18.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42ca42700 1 -- 172.21.15.7:0/1714484288 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb428099720 con 0x7fb428096e20 2023-09-30T01:55:18.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.422+0000 7fb42d243700 1 --2- 172.21.15.7:0/1714484288 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb428096a50 0x7fb428132680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:55:18.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb42ca42700 1 -- 172.21.15.7:0/1714484288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb428096e20 msgr2=0x7fb4280971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:18.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb42ca42700 1 --2- 172.21.15.7:0/1714484288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb428096e20 0x7fb4280971f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb4180098e0 tx=0x7fb4180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91282bcf0409f746 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:18.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb41ffff700 1 -- 172.21.15.7:0/1714484288 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb418011840 con 0x7fb428096e20 2023-09-30T01:55:18.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb41ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:18.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb41ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:55:18.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb41ffff700 10 monclient: dump: 2023-09-30T01:55:18.437 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:18.437 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:18.437 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:18.438 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:18.438 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:18.438 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:18.438 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:18.438 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:18.438 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:18.438 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:18.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb41ffff700 10 monclient: _finish_auth 0 2023-09-30T01:55:18.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb41ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:18.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb41ffff700 1 -- 172.21.15.7:0/1714484288 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4180119a0 con 0x7fb428096e20 2023-09-30T01:55:18.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb42f4a7700 5 monclient: authenticate success, global_id 4344 2023-09-30T01:55:18.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb42f4a7700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:55:18.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb41ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:18.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb42f4a7700 10 monclient: get_monmap_and_config success 2023-09-30T01:55:18.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb41ffff700 1 -- 172.21.15.7:0/1714484288 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb418011d40 con 0x7fb428096e20 2023-09-30T01:55:18.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb42f4a7700 4 set_mon_vals no callback set 2023-09-30T01:55:18.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb41ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:18.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb41ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:55:18.440 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb41ffff700 10 monclient: dump: 2023-09-30T01:55:18.440 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:18.440 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:18.440 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:18.440 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:18.441 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:18.441 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:18.441 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:18.441 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:18.441 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:18.441 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:18.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb42f4a7700 10 monclient: shutdown 2023-09-30T01:55:18.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb42f4a7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:18.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb42f4a7700 1 -- 172.21.15.7:0/1714484288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb428096e20 msgr2=0x7fb4280971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:18.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb42f4a7700 1 --2- 172.21.15.7:0/1714484288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb428096e20 0x7fb4280971f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb4180098e0 tx=0x7fb4180091a0 comp rx=0 tx=0).stop 2023-09-30T01:55:18.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb42f4a7700 1 -- 172.21.15.7:0/1714484288 shutdown_connections 2023-09-30T01:55:18.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb42f4a7700 1 --2- 172.21.15.7:0/1714484288 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb428096a50 0x7fb428132680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:18.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb42f4a7700 1 --2- 172.21.15.7:0/1714484288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb428096e20 0x7fb4280971f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:18.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb42f4a7700 1 -- 172.21.15.7:0/1714484288 >> 172.21.15.7:0/1714484288 conn(0x7fb428070cf0 msgr2=0x7fb4280710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:18.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb42f4a7700 1 -- 172.21.15.7:0/1714484288 shutdown_connections 2023-09-30T01:55:18.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.425+0000 7fb42f4a7700 1 -- 172.21.15.7:0/1714484288 wait complete. 2023-09-30T01:55:18.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.426+0000 7fb42f4a7700 10 monclient: build_initial_monmap 2023-09-30T01:55:18.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.426+0000 7fb42f4a7700 10 monclient: monmap: 2023-09-30T01:55:18.442 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:18.442 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:55:18.442 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:55:18.442 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:55:18.442 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:18.442 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:18.443 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:55:18.443 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:55:18.443 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:55:18.443 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:18.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.426+0000 7fb42f4a7700 1 Processor -- start 2023-09-30T01:55:18.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.426+0000 7fb42f4a7700 1 -- start start 2023-09-30T01:55:18.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.426+0000 7fb42f4a7700 10 monclient: init 2023-09-30T01:55:18.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42f4a7700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:18.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42f4a7700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:55:18.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42f4a7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb428096a50 0x7fb42812dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:18.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42f4a7700 10 monclient(hunting): picked mon.noname-b con 0x7fb428096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:55:18.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42f4a7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb428096e20 0x7fb42809db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:18.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42f4a7700 10 monclient(hunting): picked mon.noname-a con 0x7fb428096e20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:55:18.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42f4a7700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb42812ef00 0x7fb42809e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:18.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42f4a7700 10 monclient(hunting): picked mon.noname-c con 0x7fb42812ef00 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:18.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42f4a7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:18.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42f4a7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb428142a80 con 0x7fb428096e20 2023-09-30T01:55:18.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42f4a7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:18.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42f4a7700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4281334e0 con 0x7fb428096a50 2023-09-30T01:55:18.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42f4a7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:18.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42f4a7700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4280a4be0 con 0x7fb42812ef00 2023-09-30T01:55:18.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42f4a7700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:18.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42f4a7700 10 monclient(hunting): authenticate will time out at 1207.492349s 2023-09-30T01:55:18.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42ca42700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb428096e20 0x7fb42809db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:18.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42da44700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb42812ef00 0x7fb42809e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:18.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42ca42700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb428096e20 0x7fb42809db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37098/0 (socket says 172.21.15.7:37098) 2023-09-30T01:55:18.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42ca42700 1 -- 172.21.15.7:0/593798392 learned_addr learned my addr 172.21.15.7:0/593798392 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:18.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42ca42700 10 monclient(hunting): get_auth_request con 0x7fb428096e20 auth_method 0 2023-09-30T01:55:18.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42ca42700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:18.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42ca42700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:18.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42ca42700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:18.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42da44700 10 monclient(hunting): get_auth_request con 0x7fb42812ef00 auth_method 0 2023-09-30T01:55:18.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42da44700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:18.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42da44700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:18.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42da44700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:18.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42ca42700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:18.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42ca42700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:18.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42ca42700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:18.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42da44700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:18.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42da44700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:18.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42da44700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:18.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42d243700 1 --2- 172.21.15.7:0/593798392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb428096a50 0x7fb42812dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:18.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42d243700 10 monclient(hunting): get_auth_request con 0x7fb428096a50 auth_method 0 2023-09-30T01:55:18.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42d243700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:18.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42d243700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:18.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42d243700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:18.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42ca42700 10 monclient(hunting): handle_auth_done global_id 4347 payload 824 2023-09-30T01:55:18.446 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42ca42700 1 -- 172.21.15.7:0/593798392 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb42812ef00 msgr2=0x7fb42809e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:18.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42ca42700 1 --2- 172.21.15.7:0/593798392 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb42812ef00 0x7fb42809e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:18.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42ca42700 1 -- 172.21.15.7:0/593798392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb428096a50 msgr2=0x7fb42812dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:18.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42ca42700 1 --2- 172.21.15.7:0/593798392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb428096a50 0x7fb42812dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:18.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42ca42700 10 monclient: _finish_hunting 0 2023-09-30T01:55:18.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42ca42700 1 monclient: found mon.noname-a 2023-09-30T01:55:18.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42ca42700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:18.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42ca42700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:55:18.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42ca42700 1 -- 172.21.15.7:0/593798392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb424009a20 con 0x7fb428096e20 2023-09-30T01:55:18.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.427+0000 7fb42da44700 1 --2- 172.21.15.7:0/593798392 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb42812ef00 0x7fb42809e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:55:18.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb42ca42700 1 -- 172.21.15.7:0/593798392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb428096e20 msgr2=0x7fb42809db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:18.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb42ca42700 1 --2- 172.21.15.7:0/593798392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb428096e20 0x7fb42809db80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb4180093f0 tx=0x7fb418016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:18.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb41d7fa700 1 -- 172.21.15.7:0/593798392 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb418006af0 con 0x7fb428096e20 2023-09-30T01:55:18.447 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb41d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:18.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb41d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:55:18.448 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb41d7fa700 10 monclient: dump: 2023-09-30T01:55:18.448 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:18.448 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:18.448 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:18.448 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:18.448 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:18.448 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:18.448 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:18.448 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:18.448 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:18.448 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:18.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb41d7fa700 10 monclient: _finish_auth 0 2023-09-30T01:55:18.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb41d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:18.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb42f4a7700 5 monclient: authenticate success, global_id 4347 2023-09-30T01:55:18.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb41d7fa700 1 -- 172.21.15.7:0/593798392 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb418020070 con 0x7fb428096e20 2023-09-30T01:55:18.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb41d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:18.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb42f4a7700 10 monclient: _renew_subs 2023-09-30T01:55:18.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb41d7fa700 1 -- 172.21.15.7:0/593798392 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41800bdd0 con 0x7fb428096e20 2023-09-30T01:55:18.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb42f4a7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:18.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb42f4a7700 1 -- 172.21.15.7:0/593798392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb418009420 con 0x7fb428096e20 2023-09-30T01:55:18.449 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb41d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:18.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb41d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:55:18.450 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb41d7fa700 10 monclient: dump: 2023-09-30T01:55:18.450 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:18.450 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:18.450 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:18.450 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:18.450 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:18.450 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:18.450 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:18.450 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:18.451 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:18.451 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:18.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb41f7fe700 4 set_mon_vals no callback set 2023-09-30T01:55:18.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb42f4a7700 10 monclient: _renew_subs 2023-09-30T01:55:18.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb42f4a7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:18.451 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.430+0000 7fb42f4a7700 1 -- 172.21.15.7:0/593798392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4280a12d0 con 0x7fb428096e20 2023-09-30T01:55:18.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.431+0000 7fb41d7fa700 1 -- 172.21.15.7:0/593798392 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fb418022030 con 0x7fb428096e20 2023-09-30T01:55:18.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.431+0000 7fb41d7fa700 1 --2- 172.21.15.7:0/593798392 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb40c0357e0 0x7fb40c037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:18.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.431+0000 7fb41d7fa700 1 -- 172.21.15.7:0/593798392 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fb4180499e0 con 0x7fb428096e20 2023-09-30T01:55:18.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.431+0000 7fb42f4a7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:18.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.431+0000 7fb42f4a7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:18.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.431+0000 7fb42f4a7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:18.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.431+0000 7fb42f4a7700 1 -- 172.21.15.7:0/593798392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb428064c10 con 0x7fb428096e20 2023-09-30T01:55:18.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.431+0000 7fb42d243700 1 --2- 172.21.15.7:0/593798392 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb40c0357e0 0x7fb40c037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:18.452 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.431+0000 7fb42d243700 10 monclient: get_auth_request con 0x7fb40c0357e0 auth_method 0 2023-09-30T01:55:18.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.434+0000 7fb42d243700 1 --2- 172.21.15.7:0/593798392 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb40c0357e0 0x7fb40c037c90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb4240099f0 tx=0x7fb42400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:18.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.436+0000 7fb41d7fa700 1 -- 172.21.15.7:0/593798392 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb41801b660 con 0x7fb428096e20 2023-09-30T01:55:18.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.436+0000 7fb41d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:18.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.436+0000 7fb41d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:55:18.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.671+0000 7fb42f4a7700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:55:18.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.671+0000 7fb42f4a7700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:18.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.671+0000 7fb42f4a7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:18.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.671+0000 7fb42f4a7700 1 -- 172.21.15.7:0/593798392 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb428064c10 con 0x7fb428096e20 2023-09-30T01:55:18.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.671+0000 7fb41d7fa700 1 -- 172.21.15.7:0/593798392 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb4180193f0 con 0x7fb428096e20 2023-09-30T01:55:18.671 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.671+0000 7fb41d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:18.672 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.671+0000 7fb41d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:55:18.672 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:55:18.672 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:55:18.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.673+0000 7fb42f4a7700 1 -- 172.21.15.7:0/593798392 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb40c0357e0 msgr2=0x7fb40c037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:18.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.673+0000 7fb42f4a7700 1 --2- 172.21.15.7:0/593798392 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb40c0357e0 0x7fb40c037c90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb4240099f0 tx=0x7fb42400b040 comp rx=0 tx=0).stop 2023-09-30T01:55:18.674 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.673+0000 7fb42f4a7700 10 monclient: shutdown 2023-09-30T01:55:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.673+0000 7fb42f4a7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.673+0000 7fb42f4a7700 1 -- 172.21.15.7:0/593798392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb428096e20 msgr2=0x7fb42809db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.673+0000 7fb42f4a7700 1 --2- 172.21.15.7:0/593798392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb428096e20 0x7fb42809db80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb4180093f0 tx=0x7fb418016000 comp rx=0 tx=0).stop 2023-09-30T01:55:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.673+0000 7fb42f4a7700 1 -- 172.21.15.7:0/593798392 shutdown_connections 2023-09-30T01:55:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.673+0000 7fb42f4a7700 1 --2- 172.21.15.7:0/593798392 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb40c0357e0 0x7fb40c037c90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.673+0000 7fb42f4a7700 1 --2- 172.21.15.7:0/593798392 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb42812ef00 0x7fb42809e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.673+0000 7fb42f4a7700 1 --2- 172.21.15.7:0/593798392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb428096a50 0x7fb42812dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.673+0000 7fb42f4a7700 1 --2- 172.21.15.7:0/593798392 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb428096e20 0x7fb42809db80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.673+0000 7fb42f4a7700 1 -- 172.21.15.7:0/593798392 >> 172.21.15.7:0/593798392 conn(0x7fb428070cf0 msgr2=0x7fb4280a3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.673+0000 7fb42f4a7700 1 -- 172.21.15.7:0/593798392 shutdown_connections 2023-09-30T01:55:18.675 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:18.673+0000 7fb42f4a7700 1 -- 172.21.15.7:0/593798392 wait complete. 2023-09-30T01:55:21.686 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:55:21.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.817+0000 7f04e9193700 10 monclient: get_monmap_and_config 2023-09-30T01:55:21.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.817+0000 7f04e9193700 10 monclient: build_initial_monmap 2023-09-30T01:55:21.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.817+0000 7f04e9193700 10 monclient: monmap: 2023-09-30T01:55:21.818 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:21.818 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:21.819 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:55:21.818456+0000 2023-09-30T01:55:21.819 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:55:21.818456+0000 2023-09-30T01:55:21.819 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:21.819 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:21.819 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:55:21.819 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:55:21.820 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:55:21.820 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:55:21.820 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:55:21.820 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:21.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.817+0000 7f04e9193700 1 Processor -- start 2023-09-30T01:55:21.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e9193700 1 -- start start 2023-09-30T01:55:21.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e9193700 10 monclient: init 2023-09-30T01:55:21.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e9193700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:21.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e9193700 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-09-30T01:55:21.835 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e9193700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f04e4096a50 0x7f04e41327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:21.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e9193700 10 monclient(hunting): picked mon.noname-a con 0x7f04e4096a50 addr v2:172.21.15.7:3300/0 2023-09-30T01:55:21.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e9193700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f04e40a2390 0x7f04e4096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:21.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e9193700 10 monclient(hunting): picked mon.noname-c con 0x7f04e40a2390 addr v2:172.21.15.50:3300/0 2023-09-30T01:55:21.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e9193700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f04e4097360 addr v1:172.21.15.50:6789/0 2023-09-30T01:55:21.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e9193700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e40980e0 con 0x7f04e4097360 2023-09-30T01:55:21.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e9193700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:21.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e9193700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04e40991a0 con 0x7f04e4096a50 2023-09-30T01:55:21.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e9193700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:21.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e9193700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04e4099480 con 0x7f04e40a2390 2023-09-30T01:55:21.836 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e9193700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:21.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e9193700 10 monclient(hunting): authenticate will time out at 1210.883550s 2023-09-30T01:55:21.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e2d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f04e4096a50 0x7f04e41327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:21.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e2d9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f04e4096a50 0x7f04e41327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37104/0 (socket says 172.21.15.7:37104) 2023-09-30T01:55:21.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e2d9d700 1 -- 172.21.15.7:0/640299219 learned_addr learned my addr 172.21.15.7:0/640299219 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:21.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e2d9d700 10 monclient(hunting): get_auth_request con 0x7f04e4096a50 auth_method 0 2023-09-30T01:55:21.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:21.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e2d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:21.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e2d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:21.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:21.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:21.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:21.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e2d9d700 10 monclient(hunting): handle_auth_done global_id 4350 payload 274 2023-09-30T01:55:21.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e2d9d700 1 -- 172.21.15.7:0/640299219 >> v2:172.21.15.50:3300/0 conn(0x7f04e40a2390 msgr2=0x7f04e4096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:21.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e2d9d700 1 --2- 172.21.15.7:0/640299219 >> v2:172.21.15.50:3300/0 conn(0x7f04e40a2390 0x7f04e4096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:21.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e2d9d700 1 -- 172.21.15.7:0/640299219 >> v1:172.21.15.50:6789/0 conn(0x7f04e4097360 legacy=0x7f04e412ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:21.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e2d9d700 10 monclient: _finish_hunting 0 2023-09-30T01:55:21.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e2d9d700 1 monclient: found mon.noname-a 2023-09-30T01:55:21.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:21.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e2d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:55:21.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.818+0000 7f04e2d9d700 1 -- 172.21.15.7:0/640299219 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e4097bf0 con 0x7f04e4096a50 2023-09-30T01:55:21.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e2d9d700 1 -- 172.21.15.7:0/640299219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e4096a50 msgr2=0x7f04e41327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:21.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e2d9d700 1 --2- 172.21.15.7:0/640299219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e4096a50 0x7f04e41327a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f04e406a070 tx=0x7f04d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5fcd897fdf03265 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:21.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e1d9b700 1 -- 172.21.15.7:0/640299219 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d400ba30 con 0x7f04e4096a50 2023-09-30T01:55:21.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:21.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e1d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:55:21.839 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e1d9b700 10 monclient: dump: 2023-09-30T01:55:21.839 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:21.839 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:21.839 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:21.839 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:21.839 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:21.839 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:21.839 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:21.839 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:21.839 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:21.839 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:21.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e1d9b700 10 monclient: _finish_auth 0 2023-09-30T01:55:21.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e1d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:21.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e1d9b700 1 -- 172.21.15.7:0/640299219 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d400bb90 con 0x7f04e4096a50 2023-09-30T01:55:21.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e9193700 5 monclient: authenticate success, global_id 4350 2023-09-30T01:55:21.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e9193700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:55:21.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e1d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:21.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e9193700 10 monclient: get_monmap_and_config success 2023-09-30T01:55:21.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e1d9b700 1 -- 172.21.15.7:0/640299219 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d4011420 con 0x7f04e4096a50 2023-09-30T01:55:21.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e9193700 4 set_mon_vals no callback set 2023-09-30T01:55:21.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:21.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e1d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:55:21.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e1d9b700 10 monclient: dump: 2023-09-30T01:55:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:21.841 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e9193700 10 monclient: shutdown 2023-09-30T01:55:21.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e9193700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e9193700 1 -- 172.21.15.7:0/640299219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e4096a50 msgr2=0x7f04e41327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e9193700 1 --2- 172.21.15.7:0/640299219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e4096a50 0x7f04e41327a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f04e406a070 tx=0x7f04d4009250 comp rx=0 tx=0).stop 2023-09-30T01:55:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e9193700 1 -- 172.21.15.7:0/640299219 shutdown_connections 2023-09-30T01:55:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e9193700 1 --2- 172.21.15.7:0/640299219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e4096a50 0x7f04e41327a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e9193700 1 --2- 172.21.15.7:0/640299219 >> v2:172.21.15.50:3300/0 conn(0x7f04e40a2390 0x7f04e4096e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e9193700 1 -- 172.21.15.7:0/640299219 >> 172.21.15.7:0/640299219 conn(0x7f04e4070cf0 msgr2=0x7f04e40710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e9193700 1 -- 172.21.15.7:0/640299219 shutdown_connections 2023-09-30T01:55:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.821+0000 7f04e9193700 1 -- 172.21.15.7:0/640299219 wait complete. 2023-09-30T01:55:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.822+0000 7f04e9193700 10 monclient: build_initial_monmap 2023-09-30T01:55:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.822+0000 7f04e9193700 10 monclient: monmap: 2023-09-30T01:55:21.842 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:55:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:55:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:55:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:55:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:55:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:55:21.843 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.822+0000 7f04e9193700 1 Processor -- start 2023-09-30T01:55:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e9193700 1 -- start start 2023-09-30T01:55:21.843 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e9193700 10 monclient: init 2023-09-30T01:55:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e9193700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e9193700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:55:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e9193700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04e4096a50 0x7f04e412def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e9193700 10 monclient(hunting): picked mon.noname-b con 0x7f04e4096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:55:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e9193700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f04e4097360 0x7f04e409da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e9193700 10 monclient(hunting): picked mon.noname-c con 0x7f04e4097360 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e9193700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e40a2390 0x7f04e409df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e9193700 10 monclient(hunting): picked mon.noname-a con 0x7f04e40a2390 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:55:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e9193700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e9193700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e41334e0 con 0x7f04e40a2390 2023-09-30T01:55:21.844 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e9193700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e9193700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e412f0a0 con 0x7f04e4096a50 2023-09-30T01:55:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e9193700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e9193700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04e4142850 con 0x7f04e4097360 2023-09-30T01:55:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e9193700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e9193700 10 monclient(hunting): authenticate will time out at 1210.888483s 2023-09-30T01:55:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e359e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e40a2390 0x7f04e409df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e259c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f04e4097360 0x7f04e409da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e359e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e40a2390 0x7f04e409df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37116/0 (socket says 172.21.15.7:37116) 2023-09-30T01:55:21.845 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e259c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f04e4097360 0x7f04e409da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38502/0 (socket says 172.21.15.7:38502) 2023-09-30T01:55:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e359e700 1 -- 172.21.15.7:0/2400177888 learned_addr learned my addr 172.21.15.7:0/2400177888 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e359e700 10 monclient(hunting): get_auth_request con 0x7f04e40a2390 auth_method 0 2023-09-30T01:55:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e359e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e359e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e259c700 10 monclient(hunting): get_auth_request con 0x7f04e4097360 auth_method 0 2023-09-30T01:55:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e359e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:21.846 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e359e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:21.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e2d9d700 1 --2- 172.21.15.7:0/2400177888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04e4096a50 0x7f04e412def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:21.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e359e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:21.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:21.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:21.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:21.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e2d9d700 10 monclient(hunting): get_auth_request con 0x7f04e4096a50 auth_method 0 2023-09-30T01:55:21.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:21.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e2d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:21.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e2d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:21.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:21.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:21.847 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e359e700 10 monclient(hunting): handle_auth_done global_id 4353 payload 824 2023-09-30T01:55:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e359e700 1 -- 172.21.15.7:0/2400177888 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f04e4097360 msgr2=0x7f04e409da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e359e700 1 --2- 172.21.15.7:0/2400177888 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f04e4097360 0x7f04e409da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e359e700 1 -- 172.21.15.7:0/2400177888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04e4096a50 msgr2=0x7f04e412def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e359e700 1 --2- 172.21.15.7:0/2400177888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04e4096a50 0x7f04e412def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e359e700 10 monclient: _finish_hunting 0 2023-09-30T01:55:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.823+0000 7f04e359e700 1 monclient: found mon.noname-a 2023-09-30T01:55:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.824+0000 7f04e359e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.824+0000 7f04e359e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:55:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.824+0000 7f04e359e700 1 -- 172.21.15.7:0/2400177888 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d40094f0 con 0x7f04e40a2390 2023-09-30T01:55:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04e359e700 1 -- 172.21.15.7:0/2400177888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e40a2390 msgr2=0x7f04e409df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:21.848 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04e359e700 1 --2- 172.21.15.7:0/2400177888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e40a2390 0x7f04e409df90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f04dc00bbb0 tx=0x7f04dc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:21.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04caffd700 1 -- 172.21.15.7:0/2400177888 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc00c7e0 con 0x7f04e40a2390 2023-09-30T01:55:21.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04caffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:21.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04caffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:55:21.849 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04caffd700 10 monclient: dump: 2023-09-30T01:55:21.849 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:21.849 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:21.849 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:21.849 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:21.850 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:21.850 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:21.850 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:21.850 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:21.850 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:21.850 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:21.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04caffd700 10 monclient: _finish_auth 0 2023-09-30T01:55:21.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04caffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:21.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04caffd700 1 -- 172.21.15.7:0/2400177888 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04dc00c960 con 0x7f04e40a2390 2023-09-30T01:55:21.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04e9193700 5 monclient: authenticate success, global_id 4353 2023-09-30T01:55:21.850 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04e9193700 10 monclient: _renew_subs 2023-09-30T01:55:21.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04e9193700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:21.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04e9193700 1 -- 172.21.15.7:0/2400177888 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04e40a0220 con 0x7f04e40a2390 2023-09-30T01:55:21.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04caffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:21.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04caffd700 1 -- 172.21.15.7:0/2400177888 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc00cd20 con 0x7f04e40a2390 2023-09-30T01:55:21.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04caffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:21.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04caffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:55:21.851 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04caffd700 10 monclient: dump: 2023-09-30T01:55:21.851 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:21.851 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:21.851 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:21.851 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:21.851 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:21.852 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:21.852 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:21.852 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:21.852 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:21.852 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:21.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04e159a700 4 set_mon_vals no callback set 2023-09-30T01:55:21.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04e9193700 10 monclient: _renew_subs 2023-09-30T01:55:21.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04e9193700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:21.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.826+0000 7f04e9193700 1 -- 172.21.15.7:0/2400177888 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04e409fb00 con 0x7f04e40a2390 2023-09-30T01:55:21.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.827+0000 7f04caffd700 1 -- 172.21.15.7:0/2400177888 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f04dc012070 con 0x7f04e40a2390 2023-09-30T01:55:21.852 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.827+0000 7f04e9193700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:21.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.827+0000 7f04e9193700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:21.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.827+0000 7f04e9193700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:21.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.827+0000 7f04e9193700 1 -- 172.21.15.7:0/2400177888 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04e4064c10 con 0x7f04e40a2390 2023-09-30T01:55:21.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.827+0000 7f04caffd700 1 --2- 172.21.15.7:0/2400177888 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f04d00357e0 0x7f04d0037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:21.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.827+0000 7f04caffd700 1 -- 172.21.15.7:0/2400177888 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f04dc047840 con 0x7f04e40a2390 2023-09-30T01:55:21.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.827+0000 7f04e2d9d700 1 --2- 172.21.15.7:0/2400177888 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f04d00357e0 0x7f04d0037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:21.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.827+0000 7f04e2d9d700 10 monclient: get_auth_request con 0x7f04d00357e0 auth_method 0 2023-09-30T01:55:21.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.830+0000 7f04e2d9d700 1 --2- 172.21.15.7:0/2400177888 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f04d00357e0 0x7f04d0037c90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f04e406a070 tx=0x7f04d4016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:21.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.832+0000 7f04caffd700 1 -- 172.21.15.7:0/2400177888 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f04dc012730 con 0x7f04e40a2390 2023-09-30T01:55:21.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.832+0000 7f04caffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:21.853 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:21.832+0000 7f04caffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:55:22.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:22.066+0000 7f04e9193700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:55:22.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:22.066+0000 7f04e9193700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:22.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:22.066+0000 7f04e9193700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:22.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:22.066+0000 7f04e9193700 1 -- 172.21.15.7:0/2400177888 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f04e404bb40 con 0x7f04e40a2390 2023-09-30T01:55:22.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:22.067+0000 7f04caffd700 1 -- 172.21.15.7:0/2400177888 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f04dc01a070 con 0x7f04e40a2390 2023-09-30T01:55:22.067 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:22.067+0000 7f04caffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:22.068 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:22.067+0000 7f04caffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:55:22.068 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:55:22.068 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:55:22.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:22.069+0000 7f04e9193700 1 -- 172.21.15.7:0/2400177888 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f04d00357e0 msgr2=0x7f04d0037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:22.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:22.069+0000 7f04e9193700 1 --2- 172.21.15.7:0/2400177888 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f04d00357e0 0x7f04d0037c90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f04e406a070 tx=0x7f04d4016000 comp rx=0 tx=0).stop 2023-09-30T01:55:22.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:22.069+0000 7f04e9193700 10 monclient: shutdown 2023-09-30T01:55:22.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:22.069+0000 7f04e9193700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:22.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:22.069+0000 7f04e9193700 1 -- 172.21.15.7:0/2400177888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e40a2390 msgr2=0x7f04e409df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:22.070 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:22.069+0000 7f04e9193700 1 --2- 172.21.15.7:0/2400177888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e40a2390 0x7f04e409df90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f04dc00bbb0 tx=0x7f04dc009840 comp rx=0 tx=0).stop 2023-09-30T01:55:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:22.069+0000 7f04e9193700 1 -- 172.21.15.7:0/2400177888 shutdown_connections 2023-09-30T01:55:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:22.069+0000 7f04e9193700 1 --2- 172.21.15.7:0/2400177888 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f04d00357e0 0x7f04d0037c90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:22.069+0000 7f04e9193700 1 --2- 172.21.15.7:0/2400177888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04e40a2390 0x7f04e409df90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:22.069+0000 7f04e9193700 1 --2- 172.21.15.7:0/2400177888 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f04e4096a50 0x7f04e412def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:22.069+0000 7f04e9193700 1 --2- 172.21.15.7:0/2400177888 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f04e4097360 0x7f04e409da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:22.069+0000 7f04e9193700 1 -- 172.21.15.7:0/2400177888 >> 172.21.15.7:0/2400177888 conn(0x7f04e4070cf0 msgr2=0x7f04e41302b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:22.069+0000 7f04e9193700 1 -- 172.21.15.7:0/2400177888 shutdown_connections 2023-09-30T01:55:22.071 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:22.069+0000 7f04e9193700 1 -- 172.21.15.7:0/2400177888 wait complete. 2023-09-30T01:55:25.082 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:55:25.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.215+0000 7f6e77cba700 10 monclient: get_monmap_and_config 2023-09-30T01:55:25.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.215+0000 7f6e77cba700 10 monclient: build_initial_monmap 2023-09-30T01:55:25.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.216+0000 7f6e77cba700 10 monclient: monmap: 2023-09-30T01:55:25.216 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:25.216 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:25.216 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:55:25.217037+0000 2023-09-30T01:55:25.216 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:55:25.217037+0000 2023-09-30T01:55:25.216 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:25.216 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:25.217 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:55:25.217 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:55:25.217 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:55:25.217 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:55:25.217 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:55:25.217 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:25.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.216+0000 7f6e77cba700 1 Processor -- start 2023-09-30T01:55:25.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.216+0000 7f6e77cba700 1 -- start start 2023-09-30T01:55:25.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.216+0000 7f6e77cba700 10 monclient: init 2023-09-30T01:55:25.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e77cba700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:25.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e77cba700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-09-30T01:55:25.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e77cba700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e700a29e0 addr v1:172.21.15.7:6789/0 2023-09-30T01:55:25.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e77cba700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6e70097160 0x7f6e70137550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:25.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e77cba700 10 monclient(hunting): picked mon.noname-b con 0x7f6e70097160 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:25.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e77cba700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6e70097530 addr v1:172.21.15.50:6789/0 2023-09-30T01:55:25.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e77cba700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e70069cc0 con 0x7f6e700a29e0 2023-09-30T01:55:25.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e77cba700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e700a2db0 con 0x7f6e70097530 2023-09-30T01:55:25.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e77cba700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:25.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e77cba700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e70098ea0 con 0x7f6e70097160 2023-09-30T01:55:25.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e77cba700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:25.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e77cba700 10 monclient(hunting): authenticate will time out at 1214.282305s 2023-09-30T01:55:25.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e75255700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6e70097160 0x7f6e70137550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:25.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e75a56700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6e700a29e0 0x7f6e70096a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44534/0 (socket says 172.21.15.7:44534) 2023-09-30T01:55:25.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e75255700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6e70097160 0x7f6e70137550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:57560/0 (socket says 172.21.15.7:57560) 2023-09-30T01:55:25.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e75a56700 1 -- 172.21.15.7:0/12320591 learned_addr learned my addr 172.21.15.7:0/12320591 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:55:25.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e75255700 10 monclient(hunting): get_auth_request con 0x7f6e70097160 auth_method 0 2023-09-30T01:55:25.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e75255700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:25.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e75255700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:25.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e75255700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:25.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e75255700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:25.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e75255700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:25.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e75255700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:25.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e75255700 10 monclient(hunting): handle_auth_done global_id 4352 payload 274 2023-09-30T01:55:25.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e75255700 1 -- 172.21.15.7:0/12320591 >> v1:172.21.15.50:6789/0 conn(0x7f6e70097530 legacy=0x7f6e70139aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:25.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e75255700 1 -- 172.21.15.7:0/12320591 >> v1:172.21.15.7:6789/0 conn(0x7f6e700a29e0 legacy=0x7f6e70096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:25.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e75255700 10 monclient: _finish_hunting 0 2023-09-30T01:55:25.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e75255700 1 monclient: found mon.noname-b 2023-09-30T01:55:25.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e75255700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:25.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e75255700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:55:25.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.217+0000 7f6e75255700 1 -- 172.21.15.7:0/12320591 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e70098270 con 0x7f6e70097160 2023-09-30T01:55:25.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.219+0000 7f6e75255700 1 -- 172.21.15.7:0/12320591 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6e70097160 msgr2=0x7f6e70137550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:25.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.219+0000 7f6e75255700 1 --2- 172.21.15.7:0/12320591 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6e70097160 0x7f6e70137550 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c009910 tx=0x7f6e6c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62c020915c354a73 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e74a54700 1 -- 172.21.15.7:0/12320591 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c00ba30 con 0x7f6e70097160 2023-09-30T01:55:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e74a54700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e74a54700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:55:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e74a54700 10 monclient: dump: 2023-09-30T01:55:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:25.228 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:25.229 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:25.229 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:25.229 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:25.229 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:25.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e74a54700 10 monclient: _finish_auth 0 2023-09-30T01:55:25.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e74a54700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:25.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e74a54700 1 -- 172.21.15.7:0/12320591 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6c00bb90 con 0x7f6e70097160 2023-09-30T01:55:25.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e77cba700 5 monclient: authenticate success, global_id 4352 2023-09-30T01:55:25.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e77cba700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:55:25.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e74a54700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:25.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e77cba700 10 monclient: get_monmap_and_config success 2023-09-30T01:55:25.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e77cba700 4 set_mon_vals no callback set 2023-09-30T01:55:25.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e74a54700 1 -- 172.21.15.7:0/12320591 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c011420 con 0x7f6e70097160 2023-09-30T01:55:25.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e74a54700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:25.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e74a54700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:55:25.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e74a54700 10 monclient: dump: 2023-09-30T01:55:25.230 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:25.230 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:25.230 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:25.230 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:25.230 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:25.230 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:25.230 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:25.231 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:25.231 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:25.231 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:25.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e77cba700 10 monclient: shutdown 2023-09-30T01:55:25.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e77cba700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:25.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e77cba700 1 -- 172.21.15.7:0/12320591 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6e70097160 msgr2=0x7f6e70137550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:25.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e77cba700 1 --2- 172.21.15.7:0/12320591 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6e70097160 0x7f6e70137550 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c009910 tx=0x7f6e6c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:55:25.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e77cba700 1 -- 172.21.15.7:0/12320591 shutdown_connections 2023-09-30T01:55:25.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e77cba700 1 --2- 172.21.15.7:0/12320591 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6e70097160 0x7f6e70137550 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:25.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e77cba700 1 -- 172.21.15.7:0/12320591 >> 172.21.15.7:0/12320591 conn(0x7f6e700710d0 msgr2=0x7f6e700714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:25.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e77cba700 1 -- 172.21.15.7:0/12320591 shutdown_connections 2023-09-30T01:55:25.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.220+0000 7f6e77cba700 1 -- 172.21.15.7:0/12320591 wait complete. 2023-09-30T01:55:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.221+0000 7f6e77cba700 10 monclient: build_initial_monmap 2023-09-30T01:55:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.221+0000 7f6e77cba700 10 monclient: monmap: 2023-09-30T01:55:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:55:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:55:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:55:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:55:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:55:25.232 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:55:25.232 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.221+0000 7f6e77cba700 1 Processor -- start 2023-09-30T01:55:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.221+0000 7f6e77cba700 1 -- start start 2023-09-30T01:55:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.221+0000 7f6e77cba700 10 monclient: init 2023-09-30T01:55:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e77cba700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e77cba700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:55:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e77cba700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6e70097160 0x7f6e7011e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e77cba700 10 monclient(hunting): picked mon.noname-c con 0x7f6e70097160 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e77cba700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e70097530 0x7f6e7011e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e77cba700 10 monclient(hunting): picked mon.noname-a con 0x7f6e70097530 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:55:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e77cba700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e700a29e0 0x7f6e7011eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e77cba700 10 monclient(hunting): picked mon.noname-b con 0x7f6e700a29e0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:55:25.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e77cba700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:25.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e77cba700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e70121e20 con 0x7f6e70097530 2023-09-30T01:55:25.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e77cba700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:25.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e77cba700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e700a1220 con 0x7f6e700a29e0 2023-09-30T01:55:25.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e77cba700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:25.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e77cba700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e700a0620 con 0x7f6e70097160 2023-09-30T01:55:25.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e77cba700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:25.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e77cba700 10 monclient(hunting): authenticate will time out at 1214.287163s 2023-09-30T01:55:25.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75255700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e70097530 0x7f6e7011e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:25.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75a56700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6e70097160 0x7f6e7011e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:25.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75255700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e70097530 0x7f6e7011e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43606/0 (socket says 172.21.15.7:43606) 2023-09-30T01:55:25.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75a56700 10 monclient(hunting): get_auth_request con 0x7f6e70097160 auth_method 0 2023-09-30T01:55:25.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75255700 1 -- 172.21.15.7:0/2956189279 learned_addr learned my addr 172.21.15.7:0/2956189279 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:25.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75a56700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:25.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75a56700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:25.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75a56700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:25.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75255700 10 monclient(hunting): get_auth_request con 0x7f6e70097530 auth_method 0 2023-09-30T01:55:25.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75255700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:25.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75255700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:25.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75255700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:25.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75a56700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:25.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75a56700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:25.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75a56700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:25.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75255700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:25.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75255700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:25.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75255700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:25.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75a56700 10 monclient(hunting): handle_auth_done global_id 4355 payload 824 2023-09-30T01:55:25.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e76257700 1 --2- 172.21.15.7:0/2956189279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e700a29e0 0x7f6e7011eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:25.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75a56700 1 -- 172.21.15.7:0/2956189279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e700a29e0 msgr2=0x7f6e7011eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:25.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75a56700 1 --2- 172.21.15.7:0/2956189279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e700a29e0 0x7f6e7011eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:25.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75a56700 1 -- 172.21.15.7:0/2956189279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e70097530 msgr2=0x7f6e7011e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:25.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75a56700 1 --2- 172.21.15.7:0/2956189279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e70097530 0x7f6e7011e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:25.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75a56700 10 monclient: _finish_hunting 0 2023-09-30T01:55:25.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75a56700 1 monclient: found mon.noname-c 2023-09-30T01:55:25.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75a56700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:25.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75a56700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:55:25.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75a56700 1 -- 172.21.15.7:0/2956189279 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c009420 con 0x7f6e70097160 2023-09-30T01:55:25.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.222+0000 7f6e75255700 1 --2- 172.21.15.7:0/2956189279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e70097530 0x7f6e7011e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:55:25.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.224+0000 7f6e75a56700 1 -- 172.21.15.7:0/2956189279 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6e70097160 msgr2=0x7f6e7011e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:25.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.224+0000 7f6e75a56700 1 --2- 172.21.15.7:0/2956189279 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6e70097160 0x7f6e7011e430 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6e6000c510 tx=0x7f6e60007e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:25.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.224+0000 7f6e65ffb700 1 -- 172.21.15.7:0/2956189279 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e600074f0 con 0x7f6e70097160 2023-09-30T01:55:25.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.224+0000 7f6e65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:25.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.224+0000 7f6e65ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:55:25.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.224+0000 7f6e65ffb700 10 monclient: dump: 2023-09-30T01:55:25.237 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:25.238 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:25.238 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:25.238 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:25.238 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:25.238 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:25.238 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:25.238 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:25.238 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:25.238 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:25.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.224+0000 7f6e65ffb700 10 monclient: _finish_auth 0 2023-09-30T01:55:25.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.224+0000 7f6e65ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:25.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.224+0000 7f6e77cba700 5 monclient: authenticate success, global_id 4355 2023-09-30T01:55:25.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.224+0000 7f6e65ffb700 1 -- 172.21.15.7:0/2956189279 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e60007670 con 0x7f6e70097160 2023-09-30T01:55:25.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.224+0000 7f6e77cba700 10 monclient: _renew_subs 2023-09-30T01:55:25.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.224+0000 7f6e77cba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:25.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.224+0000 7f6e77cba700 1 -- 172.21.15.7:0/2956189279 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e70120d50 con 0x7f6e70097160 2023-09-30T01:55:25.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.225+0000 7f6e65ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:25.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.225+0000 7f6e65ffb700 1 -- 172.21.15.7:0/2956189279 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e60005a40 con 0x7f6e70097160 2023-09-30T01:55:25.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.225+0000 7f6e65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:25.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.225+0000 7f6e65ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:55:25.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.225+0000 7f6e65ffb700 10 monclient: dump: 2023-09-30T01:55:25.239 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:25.239 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:25.240 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:25.240 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:25.240 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:25.240 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:25.240 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:25.240 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:25.240 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:25.240 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:25.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.225+0000 7f6e67fff700 4 set_mon_vals no callback set 2023-09-30T01:55:25.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.225+0000 7f6e77cba700 10 monclient: _renew_subs 2023-09-30T01:55:25.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.225+0000 7f6e77cba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:25.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.225+0000 7f6e77cba700 1 -- 172.21.15.7:0/2956189279 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e70130bc0 con 0x7f6e70097160 2023-09-30T01:55:25.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.225+0000 7f6e65ffb700 1 -- 172.21.15.7:0/2956189279 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f6e6001b3b0 con 0x7f6e70097160 2023-09-30T01:55:25.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.225+0000 7f6e65ffb700 1 --2- 172.21.15.7:0/2956189279 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6e540357e0 0x7f6e54037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:25.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.226+0000 7f6e65ffb700 1 -- 172.21.15.7:0/2956189279 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f6e60046f90 con 0x7f6e70097160 2023-09-30T01:55:25.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.226+0000 7f6e77cba700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:25.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.226+0000 7f6e77cba700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:25.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.226+0000 7f6e77cba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:25.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.226+0000 7f6e77cba700 1 -- 172.21.15.7:0/2956189279 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e70120cf0 con 0x7f6e70097160 2023-09-30T01:55:25.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.226+0000 7f6e75255700 1 --2- 172.21.15.7:0/2956189279 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6e540357e0 0x7f6e54037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:25.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.226+0000 7f6e75255700 10 monclient: get_auth_request con 0x7f6e540357e0 auth_method 0 2023-09-30T01:55:25.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.228+0000 7f6e75255700 1 --2- 172.21.15.7:0/2956189279 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6e540357e0 0x7f6e54037c90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6e701204b0 tx=0x7f6e6c016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:25.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.230+0000 7f6e65ffb700 1 -- 172.21.15.7:0/2956189279 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6e60017020 con 0x7f6e70097160 2023-09-30T01:55:25.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.230+0000 7f6e65ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:25.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.230+0000 7f6e65ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:55:25.466 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.466+0000 7f6e77cba700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:55:25.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.466+0000 7f6e77cba700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:25.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.466+0000 7f6e77cba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:25.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.466+0000 7f6e77cba700 1 -- 172.21.15.7:0/2956189279 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6e70064c10 con 0x7f6e70097160 2023-09-30T01:55:25.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.466+0000 7f6e65ffb700 1 -- 172.21.15.7:0/2956189279 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6e60005e30 con 0x7f6e70097160 2023-09-30T01:55:25.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.466+0000 7f6e65ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:25.467 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.466+0000 7f6e65ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:55:25.467 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:55:25.467 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:55:25.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.468+0000 7f6e77cba700 1 -- 172.21.15.7:0/2956189279 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6e540357e0 msgr2=0x7f6e54037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:25.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.468+0000 7f6e77cba700 1 --2- 172.21.15.7:0/2956189279 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6e540357e0 0x7f6e54037c90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6e701204b0 tx=0x7f6e6c016000 comp rx=0 tx=0).stop 2023-09-30T01:55:25.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.468+0000 7f6e77cba700 10 monclient: shutdown 2023-09-30T01:55:25.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.468+0000 7f6e77cba700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:25.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.468+0000 7f6e77cba700 1 -- 172.21.15.7:0/2956189279 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6e70097160 msgr2=0x7f6e7011e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:25.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.468+0000 7f6e77cba700 1 --2- 172.21.15.7:0/2956189279 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6e70097160 0x7f6e7011e430 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6e6000c510 tx=0x7f6e60007e80 comp rx=0 tx=0).stop 2023-09-30T01:55:25.469 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.468+0000 7f6e77cba700 1 -- 172.21.15.7:0/2956189279 shutdown_connections 2023-09-30T01:55:25.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.468+0000 7f6e77cba700 1 --2- 172.21.15.7:0/2956189279 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6e540357e0 0x7f6e54037c90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:25.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.468+0000 7f6e77cba700 1 --2- 172.21.15.7:0/2956189279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e700a29e0 0x7f6e7011eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:25.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.468+0000 7f6e77cba700 1 --2- 172.21.15.7:0/2956189279 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6e70097160 0x7f6e7011e430 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:25.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.468+0000 7f6e77cba700 1 --2- 172.21.15.7:0/2956189279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e70097530 0x7f6e7011e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:25.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.468+0000 7f6e77cba700 1 -- 172.21.15.7:0/2956189279 >> 172.21.15.7:0/2956189279 conn(0x7f6e700710d0 msgr2=0x7f6e70099ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:25.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.469+0000 7f6e77cba700 1 -- 172.21.15.7:0/2956189279 shutdown_connections 2023-09-30T01:55:25.470 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:25.469+0000 7f6e77cba700 1 -- 172.21.15.7:0/2956189279 wait complete. 2023-09-30T01:55:28.481 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:55:28.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.612+0000 7f7e972ad700 10 monclient: get_monmap_and_config 2023-09-30T01:55:28.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.612+0000 7f7e972ad700 10 monclient: build_initial_monmap 2023-09-30T01:55:28.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.612+0000 7f7e972ad700 10 monclient: monmap: 2023-09-30T01:55:28.613 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:28.613 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:28.613 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:55:28.613937+0000 2023-09-30T01:55:28.614 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:55:28.613937+0000 2023-09-30T01:55:28.614 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:28.614 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:28.614 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:55:28.614 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:55:28.614 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:55:28.614 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:55:28.614 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:55:28.614 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:28.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.613+0000 7f7e972ad700 1 Processor -- start 2023-09-30T01:55:28.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.613+0000 7f7e972ad700 1 -- start start 2023-09-30T01:55:28.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.613+0000 7f7e972ad700 10 monclient: init 2023-09-30T01:55:28.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e972ad700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:28.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e972ad700 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-09-30T01:55:28.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e972ad700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7e90098540 addr v1:172.21.15.7:6789/0 2023-09-30T01:55:28.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e972ad700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7e901329e0 0x7f7e90137400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:28.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e972ad700 10 monclient(hunting): picked mon.noname-a con 0x7f7e901329e0 addr v2:172.21.15.7:3300/0 2023-09-30T01:55:28.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e972ad700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7e90137940 0x7f7e90139d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:28.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e972ad700 10 monclient(hunting): picked mon.noname-c con 0x7f7e90137940 addr v2:172.21.15.50:3300/0 2023-09-30T01:55:28.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e972ad700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e90069f10 con 0x7f7e90098540 2023-09-30T01:55:28.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e972ad700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:28.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e972ad700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e900a0f40 con 0x7f7e901329e0 2023-09-30T01:55:28.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e972ad700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:28.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e972ad700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e900a0790 con 0x7f7e90137940 2023-09-30T01:55:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e972ad700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e972ad700 10 monclient(hunting): authenticate will time out at 1217.679218s 2023-09-30T01:55:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e95aaa700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7e901329e0 0x7f7e90137400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e962ab700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7e90098540 0x7f7e90098910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44544/0 (socket says 172.21.15.7:44544) 2023-09-30T01:55:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e962ab700 1 -- 172.21.15.7:0/946010131 learned_addr learned my addr 172.21.15.7:0/946010131 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:55:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e95aaa700 10 monclient(hunting): get_auth_request con 0x7f7e901329e0 auth_method 0 2023-09-30T01:55:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e95aaa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e95aaa700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e95aaa700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:28.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e95aaa700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:28.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e95aaa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:28.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e95aaa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:28.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e96aac700 1 --2- 172.21.15.7:0/946010131 >> v2:172.21.15.50:3300/0 conn(0x7f7e90137940 0x7f7e90139d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:28.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e96aac700 10 monclient(hunting): get_auth_request con 0x7f7e90137940 auth_method 0 2023-09-30T01:55:28.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e96aac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:28.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e96aac700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:28.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e96aac700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:28.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e95aaa700 10 monclient(hunting): handle_auth_done global_id 4359 payload 274 2023-09-30T01:55:28.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e95aaa700 1 -- 172.21.15.7:0/946010131 >> v2:172.21.15.50:3300/0 conn(0x7f7e90137940 msgr2=0x7f7e90139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:28.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e95aaa700 1 --2- 172.21.15.7:0/946010131 >> v2:172.21.15.50:3300/0 conn(0x7f7e90137940 0x7f7e90139d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:28.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e95aaa700 1 -- 172.21.15.7:0/946010131 >> v1:172.21.15.7:6789/0 conn(0x7f7e90098540 legacy=0x7f7e90098910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:28.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e95aaa700 10 monclient: _finish_hunting 0 2023-09-30T01:55:28.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e95aaa700 1 monclient: found mon.noname-a 2023-09-30T01:55:28.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e95aaa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:28.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e95aaa700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:55:28.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e95aaa700 1 -- 172.21.15.7:0/946010131 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e90096a50 con 0x7f7e901329e0 2023-09-30T01:55:28.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.614+0000 7f7e96aac700 1 --2- 172.21.15.7:0/946010131 >> v2:172.21.15.50:3300/0 conn(0x7f7e90137940 0x7f7e90139d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:55:28.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.616+0000 7f7e95aaa700 1 -- 172.21.15.7:0/946010131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e901329e0 msgr2=0x7f7e90137400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:28.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.616+0000 7f7e95aaa700 1 --2- 172.21.15.7:0/946010131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e901329e0 0x7f7e90137400 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7e90096d30 tx=0x7f7e800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5d5c6637474615 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:28.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e952a9700 1 -- 172.21.15.7:0/946010131 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8000b880 con 0x7f7e901329e0 2023-09-30T01:55:28.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e952a9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:28.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e952a9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:55:28.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e952a9700 10 monclient: dump: 2023-09-30T01:55:28.625 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:28.625 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:28.625 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:28.625 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:28.625 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:28.625 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:28.625 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:28.625 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:28.625 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:28.625 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:28.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e952a9700 10 monclient: _finish_auth 0 2023-09-30T01:55:28.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e952a9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:28.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e952a9700 1 -- 172.21.15.7:0/946010131 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8000b9e0 con 0x7f7e901329e0 2023-09-30T01:55:28.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e972ad700 5 monclient: authenticate success, global_id 4359 2023-09-30T01:55:28.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e972ad700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:55:28.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e952a9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:28.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e972ad700 10 monclient: get_monmap_and_config success 2023-09-30T01:55:28.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e952a9700 1 -- 172.21.15.7:0/946010131 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8000bd80 con 0x7f7e901329e0 2023-09-30T01:55:28.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e972ad700 4 set_mon_vals no callback set 2023-09-30T01:55:28.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e952a9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:28.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e952a9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:55:28.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e952a9700 10 monclient: dump: 2023-09-30T01:55:28.627 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:28.627 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:28.627 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:28.627 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:28.627 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:28.627 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:28.627 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:28.627 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:28.627 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:28.628 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:28.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e972ad700 10 monclient: shutdown 2023-09-30T01:55:28.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e972ad700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:28.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e972ad700 1 -- 172.21.15.7:0/946010131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e901329e0 msgr2=0x7f7e90137400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:28.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e972ad700 1 --2- 172.21.15.7:0/946010131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e901329e0 0x7f7e90137400 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7e90096d30 tx=0x7f7e800091a0 comp rx=0 tx=0).stop 2023-09-30T01:55:28.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e972ad700 1 -- 172.21.15.7:0/946010131 shutdown_connections 2023-09-30T01:55:28.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e972ad700 1 --2- 172.21.15.7:0/946010131 >> v2:172.21.15.50:3300/0 conn(0x7f7e90137940 0x7f7e90139d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:28.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e972ad700 1 --2- 172.21.15.7:0/946010131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e901329e0 0x7f7e90137400 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:28.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e972ad700 1 -- 172.21.15.7:0/946010131 >> 172.21.15.7:0/946010131 conn(0x7f7e90071830 msgr2=0x7f7e900a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:28.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e972ad700 1 -- 172.21.15.7:0/946010131 shutdown_connections 2023-09-30T01:55:28.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.617+0000 7f7e972ad700 1 -- 172.21.15.7:0/946010131 wait complete. 2023-09-30T01:55:28.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.618+0000 7f7e972ad700 10 monclient: build_initial_monmap 2023-09-30T01:55:28.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.618+0000 7f7e972ad700 10 monclient: monmap: 2023-09-30T01:55:28.629 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:28.629 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:55:28.629 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:55:28.629 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:55:28.629 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:28.629 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:28.629 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:55:28.629 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:55:28.630 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:55:28.630 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:28.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.618+0000 7f7e972ad700 1 Processor -- start 2023-09-30T01:55:28.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.618+0000 7f7e972ad700 1 -- start start 2023-09-30T01:55:28.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.618+0000 7f7e972ad700 10 monclient: init 2023-09-30T01:55:28.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e972ad700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:28.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e972ad700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:55:28.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e972ad700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e90098540 0x7f7e9012fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:28.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e972ad700 10 monclient(hunting): picked mon.noname-b con 0x7f7e90098540 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:55:28.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e972ad700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7e901329e0 0x7f7e90133c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:28.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e972ad700 10 monclient(hunting): picked mon.noname-c con 0x7f7e901329e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:28.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e972ad700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e90137940 0x7f7e901341a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:28.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e972ad700 10 monclient(hunting): picked mon.noname-a con 0x7f7e90137940 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:55:28.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e972ad700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:28.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e972ad700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e900a34f0 con 0x7f7e90137940 2023-09-30T01:55:28.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e972ad700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:28.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e972ad700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e901327b0 con 0x7f7e90098540 2023-09-30T01:55:28.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e972ad700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:28.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e972ad700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e901364c0 con 0x7f7e901329e0 2023-09-30T01:55:28.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e972ad700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:28.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e972ad700 10 monclient(hunting): authenticate will time out at 1217.684195s 2023-09-30T01:55:28.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e96aac700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e90137940 0x7f7e901341a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:28.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e95aaa700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7e901329e0 0x7f7e90133c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:28.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e96aac700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e90137940 0x7f7e901341a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43620/0 (socket says 172.21.15.7:43620) 2023-09-30T01:55:28.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e96aac700 1 -- 172.21.15.7:0/3269247604 learned_addr learned my addr 172.21.15.7:0/3269247604 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:28.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e96aac700 10 monclient(hunting): get_auth_request con 0x7f7e90137940 auth_method 0 2023-09-30T01:55:28.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e96aac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:28.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e96aac700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:28.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e96aac700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:28.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e95aaa700 10 monclient(hunting): get_auth_request con 0x7f7e901329e0 auth_method 0 2023-09-30T01:55:28.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e95aaa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:28.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e95aaa700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:28.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e95aaa700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:28.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e962ab700 1 --2- 172.21.15.7:0/3269247604 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e90098540 0x7f7e9012fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:28.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e96aac700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:28.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e96aac700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:28.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e96aac700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:28.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e962ab700 10 monclient(hunting): get_auth_request con 0x7f7e90098540 auth_method 0 2023-09-30T01:55:28.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e962ab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:28.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e962ab700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:28.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e962ab700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:28.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e95aaa700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:28.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e95aaa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:28.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e95aaa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:28.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e962ab700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:28.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e962ab700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:28.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e962ab700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:28.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e96aac700 10 monclient(hunting): handle_auth_done global_id 4362 payload 824 2023-09-30T01:55:28.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e96aac700 1 -- 172.21.15.7:0/3269247604 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7e901329e0 msgr2=0x7f7e90133c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:28.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e96aac700 1 --2- 172.21.15.7:0/3269247604 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7e901329e0 0x7f7e90133c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:28.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e96aac700 1 -- 172.21.15.7:0/3269247604 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e90098540 msgr2=0x7f7e9012fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:28.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e96aac700 1 --2- 172.21.15.7:0/3269247604 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e90098540 0x7f7e9012fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:28.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e96aac700 10 monclient: _finish_hunting 0 2023-09-30T01:55:28.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e96aac700 1 monclient: found mon.noname-a 2023-09-30T01:55:28.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e96aac700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:28.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e96aac700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:55:28.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.619+0000 7f7e96aac700 1 -- 172.21.15.7:0/3269247604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e80009420 con 0x7f7e90137940 2023-09-30T01:55:28.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.621+0000 7f7e96aac700 1 -- 172.21.15.7:0/3269247604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e90137940 msgr2=0x7f7e901341a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:28.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.621+0000 7f7e96aac700 1 --2- 172.21.15.7:0/3269247604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e90137940 0x7f7e901341a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7e8800d180 tx=0x7f7e88009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:28.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e867fc700 1 -- 172.21.15.7:0/3269247604 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e88015440 con 0x7f7e90137940 2023-09-30T01:55:28.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:28.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e867fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:55:28.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e867fc700 10 monclient: dump: 2023-09-30T01:55:28.637 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:28.637 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:28.638 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:28.638 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:28.638 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:28.638 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:28.638 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:28.638 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:28.638 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:28.638 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:28.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e867fc700 10 monclient: _finish_auth 0 2023-09-30T01:55:28.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e867fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:28.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e867fc700 1 -- 172.21.15.7:0/3269247604 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8800f040 con 0x7f7e90137940 2023-09-30T01:55:28.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e972ad700 5 monclient: authenticate success, global_id 4362 2023-09-30T01:55:28.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e972ad700 10 monclient: _renew_subs 2023-09-30T01:55:28.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e972ad700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:28.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e972ad700 1 -- 172.21.15.7:0/3269247604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e900a05c0 con 0x7f7e90137940 2023-09-30T01:55:28.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e867fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:28.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e867fc700 1 -- 172.21.15.7:0/3269247604 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e880157f0 con 0x7f7e90137940 2023-09-30T01:55:28.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:28.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e867fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:55:28.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e867fc700 10 monclient: dump: 2023-09-30T01:55:28.639 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:28.640 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:28.640 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:28.640 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:28.640 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:28.640 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:28.640 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:28.640 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:28.640 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:28.640 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:28.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e94aa8700 4 set_mon_vals no callback set 2023-09-30T01:55:28.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e972ad700 10 monclient: _renew_subs 2023-09-30T01:55:28.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e972ad700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:28.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e972ad700 1 -- 172.21.15.7:0/3269247604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e90135880 con 0x7f7e90137940 2023-09-30T01:55:28.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e867fc700 1 -- 172.21.15.7:0/3269247604 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f7e880143a0 con 0x7f7e90137940 2023-09-30T01:55:28.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.622+0000 7f7e972ad700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:28.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.623+0000 7f7e972ad700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:28.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.623+0000 7f7e972ad700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:28.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.623+0000 7f7e972ad700 1 -- 172.21.15.7:0/3269247604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e900a2980 con 0x7f7e90137940 2023-09-30T01:55:28.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.623+0000 7f7e867fc700 1 --2- 172.21.15.7:0/3269247604 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7e74035830 0x7f7e74037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:28.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.623+0000 7f7e867fc700 1 -- 172.21.15.7:0/3269247604 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f7e88047d60 con 0x7f7e90137940 2023-09-30T01:55:28.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.623+0000 7f7e962ab700 1 --2- 172.21.15.7:0/3269247604 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7e74035830 0x7f7e74037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:28.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.623+0000 7f7e962ab700 10 monclient: get_auth_request con 0x7f7e74035830 auth_method 0 2023-09-30T01:55:28.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.626+0000 7f7e962ab700 1 --2- 172.21.15.7:0/3269247604 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7e74035830 0x7f7e74037ce0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c007880 tx=0x7f7e8c007f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:28.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.627+0000 7f7e867fc700 1 -- 172.21.15.7:0/3269247604 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7e88002690 con 0x7f7e90137940 2023-09-30T01:55:28.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.627+0000 7f7e867fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:28.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.627+0000 7f7e867fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:55:28.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.863+0000 7f7e972ad700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:55:28.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.863+0000 7f7e972ad700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:28.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.863+0000 7f7e972ad700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:28.863 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.863+0000 7f7e972ad700 1 -- 172.21.15.7:0/3269247604 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7e90064c10 con 0x7f7e90137940 2023-09-30T01:55:28.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.863+0000 7f7e867fc700 1 -- 172.21.15.7:0/3269247604 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7e88013300 con 0x7f7e90137940 2023-09-30T01:55:28.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.863+0000 7f7e867fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:28.864 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.863+0000 7f7e867fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:55:28.864 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:55:28.864 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:55:28.865 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.865+0000 7f7e972ad700 1 -- 172.21.15.7:0/3269247604 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7e74035830 msgr2=0x7f7e74037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:28.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.865+0000 7f7e972ad700 1 --2- 172.21.15.7:0/3269247604 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7e74035830 0x7f7e74037ce0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c007880 tx=0x7f7e8c007f90 comp rx=0 tx=0).stop 2023-09-30T01:55:28.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.865+0000 7f7e972ad700 10 monclient: shutdown 2023-09-30T01:55:28.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.865+0000 7f7e972ad700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:28.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.865+0000 7f7e972ad700 1 -- 172.21.15.7:0/3269247604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e90137940 msgr2=0x7f7e901341a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:28.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.865+0000 7f7e972ad700 1 --2- 172.21.15.7:0/3269247604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e90137940 0x7f7e901341a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7e8800d180 tx=0x7f7e88009a50 comp rx=0 tx=0).stop 2023-09-30T01:55:28.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.865+0000 7f7e972ad700 1 -- 172.21.15.7:0/3269247604 shutdown_connections 2023-09-30T01:55:28.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.865+0000 7f7e972ad700 1 --2- 172.21.15.7:0/3269247604 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7e74035830 0x7f7e74037ce0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:28.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.865+0000 7f7e972ad700 1 --2- 172.21.15.7:0/3269247604 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7e90137940 0x7f7e901341a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:28.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.865+0000 7f7e972ad700 1 --2- 172.21.15.7:0/3269247604 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e90098540 0x7f7e9012fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:28.866 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.865+0000 7f7e972ad700 1 --2- 172.21.15.7:0/3269247604 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7e901329e0 0x7f7e90133c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:28.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.865+0000 7f7e972ad700 1 -- 172.21.15.7:0/3269247604 >> 172.21.15.7:0/3269247604 conn(0x7f7e90071830 msgr2=0x7f7e9009b750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:28.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.865+0000 7f7e972ad700 1 -- 172.21.15.7:0/3269247604 shutdown_connections 2023-09-30T01:55:28.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:28.865+0000 7f7e972ad700 1 -- 172.21.15.7:0/3269247604 wait complete. 2023-09-30T01:55:31.878 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:55:32.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.007+0000 7f50d5393700 10 monclient: get_monmap_and_config 2023-09-30T01:55:32.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.007+0000 7f50d5393700 10 monclient: build_initial_monmap 2023-09-30T01:55:32.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.007+0000 7f50d5393700 10 monclient: monmap: 2023-09-30T01:55:32.007 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:32.007 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:32.007 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:55:32.008076+0000 2023-09-30T01:55:32.007 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:55:32.008076+0000 2023-09-30T01:55:32.007 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:32.007 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:32.008 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:55:32.008 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:55:32.008 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:55:32.008 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:55:32.008 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:55:32.008 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:32.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.007+0000 7f50d5393700 1 Processor -- start 2023-09-30T01:55:32.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.007+0000 7f50d5393700 1 -- start start 2023-09-30T01:55:32.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.007+0000 7f50d5393700 10 monclient: init 2023-09-30T01:55:32.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50d5393700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:32.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50d5393700 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-09-30T01:55:32.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50d5393700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f50d0096a50 0x7f50d01327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:32.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50d5393700 10 monclient(hunting): picked mon.noname-c con 0x7f50d0096a50 addr v2:172.21.15.50:3300/0 2023-09-30T01:55:32.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50d5393700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f50d00a2390 0x7f50d0096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:32.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50d5393700 10 monclient(hunting): picked mon.noname-a con 0x7f50d00a2390 addr v2:172.21.15.7:3300/0 2023-09-30T01:55:32.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50d5393700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50d0097360 0x7f50d012ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:32.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50d5393700 10 monclient(hunting): picked mon.noname-b con 0x7f50d0097360 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:32.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50d5393700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:32.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50d5393700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50d00980e0 con 0x7f50d00a2390 2023-09-30T01:55:32.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50d5393700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:32.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50d5393700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50d00991a0 con 0x7f50d0096a50 2023-09-30T01:55:32.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50d5393700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:32.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50d5393700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d0099480 con 0x7f50d0097360 2023-09-30T01:55:32.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50d5393700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50d5393700 10 monclient(hunting): authenticate will time out at 1221.073119s 2023-09-30T01:55:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50ce7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f50d00a2390 0x7f50d0096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50cf7fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50d0097360 0x7f50d012ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50cf7fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50d0097360 0x7f50d012ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:57568/0 (socket says 172.21.15.7:57568) 2023-09-30T01:55:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50ce7fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f50d00a2390 0x7f50d0096e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43626/0 (socket says 172.21.15.7:43626) 2023-09-30T01:55:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50cf7fe700 1 -- 172.21.15.7:0/2043704780 learned_addr learned my addr 172.21.15.7:0/2043704780 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50cf7fe700 10 monclient(hunting): get_auth_request con 0x7f50d0097360 auth_method 0 2023-09-30T01:55:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50cf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50cf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50cf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50ce7fc700 10 monclient(hunting): get_auth_request con 0x7f50d00a2390 auth_method 0 2023-09-30T01:55:32.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50ce7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:32.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50ce7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:32.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50ce7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:32.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50cf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:32.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50cf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:32.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50cf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:32.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50ce7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:32.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50ce7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:32.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50ce7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:32.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50ceffd700 1 --2- 172.21.15.7:0/2043704780 >> v2:172.21.15.50:3300/0 conn(0x7f50d0096a50 0x7f50d01327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:32.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50ceffd700 10 monclient(hunting): get_auth_request con 0x7f50d0096a50 auth_method 0 2023-09-30T01:55:32.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50ceffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:32.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50ceffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:32.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50ceffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:32.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50cf7fe700 10 monclient(hunting): handle_auth_done global_id 4361 payload 274 2023-09-30T01:55:32.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50cf7fe700 1 -- 172.21.15.7:0/2043704780 >> v2:172.21.15.50:3300/0 conn(0x7f50d0096a50 msgr2=0x7f50d01327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:32.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50cf7fe700 1 --2- 172.21.15.7:0/2043704780 >> v2:172.21.15.50:3300/0 conn(0x7f50d0096a50 0x7f50d01327a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:32.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50cf7fe700 1 -- 172.21.15.7:0/2043704780 >> v2:172.21.15.7:3300/0 conn(0x7f50d00a2390 msgr2=0x7f50d0096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:32.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50cf7fe700 1 --2- 172.21.15.7:0/2043704780 >> v2:172.21.15.7:3300/0 conn(0x7f50d00a2390 0x7f50d0096e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:32.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50cf7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:55:32.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50cf7fe700 1 monclient: found mon.noname-b 2023-09-30T01:55:32.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50cf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:32.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50cf7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:55:32.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50cf7fe700 1 -- 172.21.15.7:0/2043704780 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d0097bf0 con 0x7f50d0097360 2023-09-30T01:55:32.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.008+0000 7f50ce7fc700 1 --2- 172.21.15.7:0/2043704780 >> v2:172.21.15.7:3300/0 conn(0x7f50d00a2390 0x7f50d0096e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:55:32.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.010+0000 7f50cf7fe700 1 -- 172.21.15.7:0/2043704780 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50d0097360 msgr2=0x7f50d012ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:32.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.010+0000 7f50cf7fe700 1 --2- 172.21.15.7:0/2043704780 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50d0097360 0x7f50d012ef00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f50d00a2b00 tx=0x7f50c8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dd6a420b6091745 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:32.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.010+0000 7f50cdffb700 1 -- 172.21.15.7:0/2043704780 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c800e070 con 0x7f50d0097360 2023-09-30T01:55:32.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50cdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:32.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50cdffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:55:32.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50cdffb700 10 monclient: dump: 2023-09-30T01:55:32.020 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:32.020 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:32.020 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:32.020 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:32.020 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:32.020 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:32.021 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50cdffb700 10 monclient: _finish_auth 0 2023-09-30T01:55:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50cdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50d5393700 5 monclient: authenticate success, global_id 4361 2023-09-30T01:55:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50d5393700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:55:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50cdffb700 1 -- 172.21.15.7:0/2043704780 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c800bc30 con 0x7f50d0097360 2023-09-30T01:55:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50cdffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:32.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50d5393700 10 monclient: get_monmap_and_config success 2023-09-30T01:55:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50cdffb700 1 -- 172.21.15.7:0/2043704780 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c80092e0 con 0x7f50d0097360 2023-09-30T01:55:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50cdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50d5393700 4 set_mon_vals no callback set 2023-09-30T01:55:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50cdffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:55:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50cdffb700 10 monclient: dump: 2023-09-30T01:55:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:32.022 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:32.023 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:32.023 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:32.023 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:32.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50d5393700 10 monclient: shutdown 2023-09-30T01:55:32.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50d5393700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:32.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50d5393700 1 -- 172.21.15.7:0/2043704780 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50d0097360 msgr2=0x7f50d012ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:32.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50d5393700 1 --2- 172.21.15.7:0/2043704780 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50d0097360 0x7f50d012ef00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f50d00a2b00 tx=0x7f50c8009840 comp rx=0 tx=0).stop 2023-09-30T01:55:32.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50d5393700 1 -- 172.21.15.7:0/2043704780 shutdown_connections 2023-09-30T01:55:32.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50d5393700 1 --2- 172.21.15.7:0/2043704780 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50d0097360 0x7f50d012ef00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:32.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50d5393700 1 --2- 172.21.15.7:0/2043704780 >> v2:172.21.15.50:3300/0 conn(0x7f50d0096a50 0x7f50d01327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:32.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50d5393700 1 --2- 172.21.15.7:0/2043704780 >> v2:172.21.15.7:3300/0 conn(0x7f50d00a2390 0x7f50d0096e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:32.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50d5393700 1 -- 172.21.15.7:0/2043704780 >> 172.21.15.7:0/2043704780 conn(0x7f50d0070cf0 msgr2=0x7f50d00710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:32.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50d5393700 1 -- 172.21.15.7:0/2043704780 shutdown_connections 2023-09-30T01:55:32.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.011+0000 7f50d5393700 1 -- 172.21.15.7:0/2043704780 wait complete. 2023-09-30T01:55:32.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.012+0000 7f50d5393700 10 monclient: build_initial_monmap 2023-09-30T01:55:32.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.012+0000 7f50d5393700 10 monclient: monmap: 2023-09-30T01:55:32.024 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:32.024 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:55:32.024 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:55:32.024 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:55:32.024 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:32.024 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:32.024 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:55:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:55:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:55:32.025 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.012+0000 7f50d5393700 1 Processor -- start 2023-09-30T01:55:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.012+0000 7f50d5393700 1 -- start start 2023-09-30T01:55:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.012+0000 7f50d5393700 10 monclient: init 2023-09-30T01:55:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50d5393700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50d5393700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:55:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50d5393700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50d0096a50 0x7f50d00a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50d5393700 10 monclient(hunting): picked mon.noname-b con 0x7f50d0096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:55:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50d5393700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50d0097360 0x7f50d012dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:32.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50d5393700 10 monclient(hunting): picked mon.noname-c con 0x7f50d0097360 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:32.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50d5393700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50d00a2390 0x7f50d012e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:32.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50d5393700 10 monclient(hunting): picked mon.noname-a con 0x7f50d00a2390 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:55:32.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50d5393700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:32.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50d5393700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d011dbe0 con 0x7f50d00a2390 2023-09-30T01:55:32.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50d5393700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:32.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50d5393700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d012eed0 con 0x7f50d0096a50 2023-09-30T01:55:32.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50d5393700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:32.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50d5393700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d011ded0 con 0x7f50d0097360 2023-09-30T01:55:32.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50d5393700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:32.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50d5393700 10 monclient(hunting): authenticate will time out at 1221.078267s 2023-09-30T01:55:32.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50ce7fc700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50d0097360 0x7f50d012dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:32.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50cf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50d00a2390 0x7f50d012e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:32.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50ce7fc700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50d0097360 0x7f50d012dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:57578/0 (socket says 172.21.15.7:57578) 2023-09-30T01:55:32.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50ce7fc700 1 -- 172.21.15.7:0/3117654386 learned_addr learned my addr 172.21.15.7:0/3117654386 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:32.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50cf7fe700 10 monclient(hunting): get_auth_request con 0x7f50d00a2390 auth_method 0 2023-09-30T01:55:32.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50cf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:32.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50cf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:32.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50cf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:32.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50ce7fc700 10 monclient(hunting): get_auth_request con 0x7f50d0097360 auth_method 0 2023-09-30T01:55:32.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50ce7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:32.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50ce7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:32.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50ce7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:32.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50cf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50cf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50ceffd700 1 --2- 172.21.15.7:0/3117654386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50d0096a50 0x7f50d00a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50cf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50ce7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50ce7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50ce7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50ceffd700 10 monclient(hunting): get_auth_request con 0x7f50d0096a50 auth_method 0 2023-09-30T01:55:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50ceffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50ceffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50ceffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50ceffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50ceffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:32.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50ceffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:32.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50cf7fe700 10 monclient(hunting): handle_auth_done global_id 4368 payload 824 2023-09-30T01:55:32.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50cf7fe700 1 -- 172.21.15.7:0/3117654386 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50d0097360 msgr2=0x7f50d012dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:32.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50cf7fe700 1 --2- 172.21.15.7:0/3117654386 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50d0097360 0x7f50d012dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:32.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50cf7fe700 1 -- 172.21.15.7:0/3117654386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50d0096a50 msgr2=0x7f50d00a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:32.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50cf7fe700 1 --2- 172.21.15.7:0/3117654386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50d0096a50 0x7f50d00a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:32.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50cf7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:55:32.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50cf7fe700 1 monclient: found mon.noname-a 2023-09-30T01:55:32.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50cf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:32.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50cf7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:55:32.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50cf7fe700 1 -- 172.21.15.7:0/3117654386 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c4009420 con 0x7f50d00a2390 2023-09-30T01:55:32.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.013+0000 7f50ce7fc700 1 --2- 172.21.15.7:0/3117654386 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50d0097360 0x7f50d012dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:55:32.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.015+0000 7f50cf7fe700 1 -- 172.21.15.7:0/3117654386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50d00a2390 msgr2=0x7f50d012e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:32.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.015+0000 7f50cf7fe700 1 --2- 172.21.15.7:0/3117654386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50d00a2390 0x7f50d012e2c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f50d006a070 tx=0x7f50c8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:32.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50b6ffd700 1 -- 172.21.15.7:0/3117654386 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c8007bb0 con 0x7f50d00a2390 2023-09-30T01:55:32.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50b6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:32.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50b6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:55:32.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50b6ffd700 10 monclient: dump: 2023-09-30T01:55:32.033 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:32.033 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:32.033 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:32.033 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:32.033 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:32.033 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:32.033 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:32.033 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:32.033 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:32.033 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:32.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50b6ffd700 10 monclient: _finish_auth 0 2023-09-30T01:55:32.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50b6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:32.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50b6ffd700 1 -- 172.21.15.7:0/3117654386 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c8007d10 con 0x7f50d00a2390 2023-09-30T01:55:32.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50d5393700 5 monclient: authenticate success, global_id 4368 2023-09-30T01:55:32.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50d5393700 10 monclient: _renew_subs 2023-09-30T01:55:32.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50d5393700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:32.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50d5393700 1 -- 172.21.15.7:0/3117654386 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c8009ae0 con 0x7f50d00a2390 2023-09-30T01:55:32.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50b6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:32.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50b6ffd700 1 -- 172.21.15.7:0/3117654386 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c80266b0 con 0x7f50d00a2390 2023-09-30T01:55:32.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50b6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:32.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50b6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:55:32.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50b6ffd700 10 monclient: dump: 2023-09-30T01:55:32.034 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:32.034 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:32.035 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:32.035 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:32.035 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:32.035 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:32.035 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:32.035 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:32.035 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:32.035 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:32.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50cd7fa700 4 set_mon_vals no callback set 2023-09-30T01:55:32.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50d5393700 10 monclient: _renew_subs 2023-09-30T01:55:32.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50d5393700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:32.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50d5393700 1 -- 172.21.15.7:0/3117654386 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d011e490 con 0x7f50d00a2390 2023-09-30T01:55:32.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50b6ffd700 1 -- 172.21.15.7:0/3117654386 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f50c801d420 con 0x7f50d00a2390 2023-09-30T01:55:32.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50d5393700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:32.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50d5393700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:32.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50d5393700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:32.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.016+0000 7f50d5393700 1 -- 172.21.15.7:0/3117654386 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50d0064c10 con 0x7f50d00a2390 2023-09-30T01:55:32.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.017+0000 7f50b6ffd700 1 --2- 172.21.15.7:0/3117654386 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f50bc0357e0 0x7f50bc037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:32.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.017+0000 7f50b6ffd700 1 -- 172.21.15.7:0/3117654386 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f50c8049b40 con 0x7f50d00a2390 2023-09-30T01:55:32.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.017+0000 7f50ceffd700 1 --2- 172.21.15.7:0/3117654386 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f50bc0357e0 0x7f50bc037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:32.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.017+0000 7f50ceffd700 10 monclient: get_auth_request con 0x7f50bc0357e0 auth_method 0 2023-09-30T01:55:32.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.020+0000 7f50ceffd700 1 --2- 172.21.15.7:0/3117654386 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f50bc0357e0 0x7f50bc037c90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f50c0006fd0 tx=0x7f50c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:32.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.021+0000 7f50b6ffd700 1 -- 172.21.15.7:0/3117654386 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f50c80117a0 con 0x7f50d00a2390 2023-09-30T01:55:32.036 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.021+0000 7f50b6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:32.037 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.021+0000 7f50b6ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:55:32.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.254+0000 7f50d5393700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:55:32.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.254+0000 7f50d5393700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:32.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.254+0000 7f50d5393700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:32.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.254+0000 7f50d5393700 1 -- 172.21.15.7:0/3117654386 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f50d004bb40 con 0x7f50d00a2390 2023-09-30T01:55:32.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.255+0000 7f50b6ffd700 1 -- 172.21.15.7:0/3117654386 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f50c8021020 con 0x7f50d00a2390 2023-09-30T01:55:32.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.255+0000 7f50b6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:32.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.255+0000 7f50b6ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:55:32.256 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:55:32.256 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:55:32.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.257+0000 7f50d5393700 1 -- 172.21.15.7:0/3117654386 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f50bc0357e0 msgr2=0x7f50bc037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:32.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.257+0000 7f50d5393700 1 --2- 172.21.15.7:0/3117654386 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f50bc0357e0 0x7f50bc037c90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f50c0006fd0 tx=0x7f50c0008040 comp rx=0 tx=0).stop 2023-09-30T01:55:32.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.257+0000 7f50d5393700 10 monclient: shutdown 2023-09-30T01:55:32.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.257+0000 7f50d5393700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:32.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.257+0000 7f50d5393700 1 -- 172.21.15.7:0/3117654386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50d00a2390 msgr2=0x7f50d012e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:32.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.257+0000 7f50d5393700 1 --2- 172.21.15.7:0/3117654386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50d00a2390 0x7f50d012e2c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f50d006a070 tx=0x7f50c8018000 comp rx=0 tx=0).stop 2023-09-30T01:55:32.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.257+0000 7f50d5393700 1 -- 172.21.15.7:0/3117654386 shutdown_connections 2023-09-30T01:55:32.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.257+0000 7f50d5393700 1 --2- 172.21.15.7:0/3117654386 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f50bc0357e0 0x7f50bc037c90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:32.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.257+0000 7f50d5393700 1 --2- 172.21.15.7:0/3117654386 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50d00a2390 0x7f50d012e2c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:32.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.257+0000 7f50d5393700 1 --2- 172.21.15.7:0/3117654386 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50d0096a50 0x7f50d00a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:32.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.257+0000 7f50d5393700 1 --2- 172.21.15.7:0/3117654386 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50d0097360 0x7f50d012dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:32.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.257+0000 7f50d5393700 1 -- 172.21.15.7:0/3117654386 >> 172.21.15.7:0/3117654386 conn(0x7f50d0070cf0 msgr2=0x7f50d00a3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:32.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.257+0000 7f50d5393700 1 -- 172.21.15.7:0/3117654386 shutdown_connections 2023-09-30T01:55:32.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:32.257+0000 7f50d5393700 1 -- 172.21.15.7:0/3117654386 wait complete. 2023-09-30T01:55:35.270 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:55:35.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.401+0000 7f6f45b66700 10 monclient: get_monmap_and_config 2023-09-30T01:55:35.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.401+0000 7f6f45b66700 10 monclient: build_initial_monmap 2023-09-30T01:55:35.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.401+0000 7f6f45b66700 10 monclient: monmap: 2023-09-30T01:55:35.402 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:35.402 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:35.402 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:55:35.402749+0000 2023-09-30T01:55:35.402 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:55:35.402749+0000 2023-09-30T01:55:35.402 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:35.402 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:35.402 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:55:35.403 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:55:35.403 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:55:35.403 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:55:35.403 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:55:35.403 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:35.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.402+0000 7f6f45b66700 1 Processor -- start 2023-09-30T01:55:35.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.402+0000 7f6f45b66700 1 -- start start 2023-09-30T01:55:35.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.402+0000 7f6f45b66700 10 monclient: init 2023-09-30T01:55:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.402+0000 7f6f45b66700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.402+0000 7f6f45b66700 10 monclient: _add_conns ranks=[0,1,2,4,3] 2023-09-30T01:55:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f45b66700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6f40096a50 addr v1:172.21.15.7:6789/0 2023-09-30T01:55:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f45b66700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6f400a2390 addr v1:172.21.15.50:6789/0 2023-09-30T01:55:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f45b66700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f40097530 0x7f6f4013a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f45b66700 10 monclient(hunting): picked mon.noname-b con 0x7f6f40097530 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f45b66700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f400980e0 con 0x7f6f40096a50 2023-09-30T01:55:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f45b66700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f40132eb0 con 0x7f6f400a2390 2023-09-30T01:55:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f45b66700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:35.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f45b66700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f400991a0 con 0x7f6f40097530 2023-09-30T01:55:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f45b66700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f45b66700 10 monclient(hunting): authenticate will time out at 1224.468124s 2023-09-30T01:55:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f3f7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6f40096a50 0x7f6f401327a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45952/0 (socket says 172.21.15.7:45952) 2023-09-30T01:55:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f3f7fe700 1 -- 172.21.15.7:0/2410733051 learned_addr learned my addr 172.21.15.7:0/2410733051 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:55:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f3ffff700 1 --2- 172.21.15.7:0/2410733051 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f40097530 0x7f6f4013a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f3ffff700 10 monclient(hunting): get_auth_request con 0x7f6f40097530 auth_method 0 2023-09-30T01:55:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f3ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f3ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:35.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f3ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:35.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f3ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:35.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f3ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:35.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f3ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:35.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f3ffff700 10 monclient(hunting): handle_auth_done global_id 4367 payload 274 2023-09-30T01:55:35.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f3ffff700 1 -- 172.21.15.7:0/2410733051 >> v1:172.21.15.50:6789/0 conn(0x7f6f400a2390 legacy=0x7f6f40096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:35.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f3ffff700 1 -- 172.21.15.7:0/2410733051 >> v1:172.21.15.7:6789/0 conn(0x7f6f40096a50 legacy=0x7f6f401327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:35.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f3ffff700 10 monclient: _finish_hunting 0 2023-09-30T01:55:35.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f3ffff700 1 monclient: found mon.noname-b 2023-09-30T01:55:35.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f3ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:35.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f3ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:55:35.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.403+0000 7f6f3ffff700 1 -- 172.21.15.7:0/2410733051 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f40099480 con 0x7f6f40097530 2023-09-30T01:55:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.405+0000 7f6f3ffff700 1 -- 172.21.15.7:0/2410733051 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f40097530 msgr2=0x7f6f4013a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.405+0000 7f6f3ffff700 1 --2- 172.21.15.7:0/2410733051 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f40097530 0x7f6f4013a730 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6f38009e90 tx=0x7f6f38009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81680a7c2572cc20 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.405+0000 7f6f3e7fc700 1 -- 172.21.15.7:0/2410733051 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3800e070 con 0x7f6f40097530 2023-09-30T01:55:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f3e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f3e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:55:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f3e7fc700 10 monclient: dump: 2023-09-30T01:55:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:35.417 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:35.418 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f3e7fc700 10 monclient: _finish_auth 0 2023-09-30T01:55:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f3e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f3e7fc700 1 -- 172.21.15.7:0/2410733051 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f38012430 con 0x7f6f40097530 2023-09-30T01:55:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f45b66700 5 monclient: authenticate success, global_id 4367 2023-09-30T01:55:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f45b66700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:55:35.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f3e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f45b66700 10 monclient: get_monmap_and_config success 2023-09-30T01:55:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f3e7fc700 1 -- 172.21.15.7:0/2410733051 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3800bea0 con 0x7f6f40097530 2023-09-30T01:55:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f3e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f45b66700 4 set_mon_vals no callback set 2023-09-30T01:55:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f3e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:55:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f3e7fc700 10 monclient: dump: 2023-09-30T01:55:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:35.419 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:35.420 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f45b66700 10 monclient: shutdown 2023-09-30T01:55:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f45b66700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f45b66700 1 -- 172.21.15.7:0/2410733051 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f40097530 msgr2=0x7f6f4013a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f45b66700 1 --2- 172.21.15.7:0/2410733051 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f40097530 0x7f6f4013a730 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6f38009e90 tx=0x7f6f38009780 comp rx=0 tx=0).stop 2023-09-30T01:55:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f45b66700 1 -- 172.21.15.7:0/2410733051 shutdown_connections 2023-09-30T01:55:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f45b66700 1 --2- 172.21.15.7:0/2410733051 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f40097530 0x7f6f4013a730 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f45b66700 1 -- 172.21.15.7:0/2410733051 >> 172.21.15.7:0/2410733051 conn(0x7f6f40070cf0 msgr2=0x7f6f400710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:35.420 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f45b66700 1 -- 172.21.15.7:0/2410733051 shutdown_connections 2023-09-30T01:55:35.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.406+0000 7f6f45b66700 1 -- 172.21.15.7:0/2410733051 wait complete. 2023-09-30T01:55:35.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.407+0000 7f6f45b66700 10 monclient: build_initial_monmap 2023-09-30T01:55:35.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.407+0000 7f6f45b66700 10 monclient: monmap: 2023-09-30T01:55:35.421 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:35.421 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:55:35.421 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:55:35.421 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:55:35.421 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:35.421 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:35.421 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:55:35.422 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:55:35.422 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:55:35.422 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:35.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.407+0000 7f6f45b66700 1 Processor -- start 2023-09-30T01:55:35.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.407+0000 7f6f45b66700 1 -- start start 2023-09-30T01:55:35.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.407+0000 7f6f45b66700 10 monclient: init 2023-09-30T01:55:35.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f45b66700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:35.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f45b66700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:55:35.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f45b66700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f40096a50 0x7f6f400a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:35.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f45b66700 10 monclient(hunting): picked mon.noname-b con 0x7f6f40096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:55:35.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f45b66700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f40097530 0x7f6f4009a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:35.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f45b66700 10 monclient(hunting): picked mon.noname-a con 0x7f6f40097530 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:55:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f45b66700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f400a2390 0x7f6f4009a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f45b66700 10 monclient(hunting): picked mon.noname-c con 0x7f6f400a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f45b66700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f45b66700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4009e130 con 0x7f6f40097530 2023-09-30T01:55:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f45b66700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f45b66700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f401334e0 con 0x7f6f40096a50 2023-09-30T01:55:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f45b66700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f45b66700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f400a5230 con 0x7f6f400a2390 2023-09-30T01:55:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f45b66700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f45b66700 10 monclient(hunting): authenticate will time out at 1224.473200s 2023-09-30T01:55:35.423 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f40097530 0x7f6f4009a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3ffff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f400a2390 0x7f6f4009a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3ffff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f400a2390 0x7f6f4009a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:50608/0 (socket says 172.21.15.7:50608) 2023-09-30T01:55:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3effd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f40097530 0x7f6f4009a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55840/0 (socket says 172.21.15.7:55840) 2023-09-30T01:55:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3ffff700 1 -- 172.21.15.7:0/843940971 learned_addr learned my addr 172.21.15.7:0/843940971 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3ffff700 10 monclient(hunting): get_auth_request con 0x7f6f400a2390 auth_method 0 2023-09-30T01:55:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3effd700 10 monclient(hunting): get_auth_request con 0x7f6f40097530 auth_method 0 2023-09-30T01:55:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:35.424 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3f7fe700 1 --2- 172.21.15.7:0/843940971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f40096a50 0x7f6f400a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3f7fe700 10 monclient(hunting): get_auth_request con 0x7f6f40096a50 auth_method 0 2023-09-30T01:55:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:35.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3effd700 10 monclient(hunting): handle_auth_done global_id 4371 payload 824 2023-09-30T01:55:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3effd700 1 -- 172.21.15.7:0/843940971 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f400a2390 msgr2=0x7f6f4009a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3effd700 1 --2- 172.21.15.7:0/843940971 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f400a2390 0x7f6f4009a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3effd700 1 -- 172.21.15.7:0/843940971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f40096a50 msgr2=0x7f6f400a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3effd700 1 --2- 172.21.15.7:0/843940971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f40096a50 0x7f6f400a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3effd700 10 monclient: _finish_hunting 0 2023-09-30T01:55:35.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3effd700 1 monclient: found mon.noname-a 2023-09-30T01:55:35.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:35.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:55:35.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3effd700 1 -- 172.21.15.7:0/843940971 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f38009a20 con 0x7f6f40097530 2023-09-30T01:55:35.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.408+0000 7f6f3ffff700 1 --2- 172.21.15.7:0/843940971 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f400a2390 0x7f6f4009a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:55:35.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.410+0000 7f6f3effd700 1 -- 172.21.15.7:0/843940971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f40097530 msgr2=0x7f6f4009a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:35.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.410+0000 7f6f3effd700 1 --2- 172.21.15.7:0/843940971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f40097530 0x7f6f4009a490 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400c510 tx=0x7f6f34007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:35.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.411+0000 7f6f277fe700 1 -- 172.21.15.7:0/843940971 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3400ce80 con 0x7f6f40097530 2023-09-30T01:55:35.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.411+0000 7f6f277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:35.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.411+0000 7f6f277fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:55:35.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.411+0000 7f6f277fe700 10 monclient: dump: 2023-09-30T01:55:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:35.428 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:35.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.411+0000 7f6f277fe700 10 monclient: _finish_auth 0 2023-09-30T01:55:35.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.411+0000 7f6f277fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:35.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.411+0000 7f6f277fe700 1 -- 172.21.15.7:0/843940971 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f340117a0 con 0x7f6f40097530 2023-09-30T01:55:35.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.411+0000 7f6f45b66700 5 monclient: authenticate success, global_id 4371 2023-09-30T01:55:35.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.411+0000 7f6f45b66700 10 monclient: _renew_subs 2023-09-30T01:55:35.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.411+0000 7f6f45b66700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:35.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.411+0000 7f6f45b66700 1 -- 172.21.15.7:0/843940971 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f4009e7a0 con 0x7f6f40097530 2023-09-30T01:55:35.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.411+0000 7f6f277fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:35.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.411+0000 7f6f277fe700 1 -- 172.21.15.7:0/843940971 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f34011b60 con 0x7f6f40097530 2023-09-30T01:55:35.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.411+0000 7f6f277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:35.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.411+0000 7f6f277fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:55:35.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.411+0000 7f6f277fe700 10 monclient: dump: 2023-09-30T01:55:35.430 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:35.430 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:35.430 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:35.430 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:35.430 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:35.430 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:35.430 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:35.430 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:35.430 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:35.430 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:35.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.411+0000 7f6f3dffb700 4 set_mon_vals no callback set 2023-09-30T01:55:35.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.411+0000 7f6f45b66700 10 monclient: _renew_subs 2023-09-30T01:55:35.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.411+0000 7f6f45b66700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:35.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.411+0000 7f6f45b66700 1 -- 172.21.15.7:0/843940971 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f400a1af0 con 0x7f6f40097530 2023-09-30T01:55:35.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.412+0000 7f6f277fe700 1 -- 172.21.15.7:0/843940971 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f6f3401c460 con 0x7f6f40097530 2023-09-30T01:55:35.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.412+0000 7f6f45b66700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:35.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.412+0000 7f6f45b66700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:35.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.412+0000 7f6f45b66700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:35.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.412+0000 7f6f45b66700 1 -- 172.21.15.7:0/843940971 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f4009df90 con 0x7f6f40097530 2023-09-30T01:55:35.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.412+0000 7f6f277fe700 1 --2- 172.21.15.7:0/843940971 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6f2c0357e0 0x7f6f2c037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:35.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.412+0000 7f6f277fe700 1 -- 172.21.15.7:0/843940971 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f6f34047850 con 0x7f6f40097530 2023-09-30T01:55:35.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.412+0000 7f6f3f7fe700 1 --2- 172.21.15.7:0/843940971 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6f2c0357e0 0x7f6f2c037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:35.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.412+0000 7f6f3f7fe700 10 monclient: get_auth_request con 0x7f6f2c0357e0 auth_method 0 2023-09-30T01:55:35.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.415+0000 7f6f3f7fe700 1 --2- 172.21.15.7:0/843940971 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6f2c0357e0 0x7f6f2c037c90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6f30007880 tx=0x7f6f30007f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:35.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.417+0000 7f6f277fe700 1 -- 172.21.15.7:0/843940971 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6f4009df90 con 0x7f6f40097530 2023-09-30T01:55:35.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.417+0000 7f6f277fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:35.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.417+0000 7f6f277fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:55:35.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.653+0000 7f6f45b66700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:55:35.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.653+0000 7f6f45b66700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:35.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.653+0000 7f6f45b66700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:35.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.653+0000 7f6f45b66700 1 -- 172.21.15.7:0/843940971 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6f4009df90 con 0x7f6f40097530 2023-09-30T01:55:35.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.654+0000 7f6f277fe700 1 -- 172.21.15.7:0/843940971 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6f4009df90 con 0x7f6f40097530 2023-09-30T01:55:35.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.654+0000 7f6f277fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:35.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.654+0000 7f6f277fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:55:35.654 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:55:35.655 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:55:35.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.655+0000 7f6f45b66700 1 -- 172.21.15.7:0/843940971 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6f2c0357e0 msgr2=0x7f6f2c037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:35.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.656+0000 7f6f45b66700 1 --2- 172.21.15.7:0/843940971 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6f2c0357e0 0x7f6f2c037c90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6f30007880 tx=0x7f6f30007f90 comp rx=0 tx=0).stop 2023-09-30T01:55:35.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.656+0000 7f6f45b66700 10 monclient: shutdown 2023-09-30T01:55:35.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.656+0000 7f6f45b66700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:35.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.656+0000 7f6f45b66700 1 -- 172.21.15.7:0/843940971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f40097530 msgr2=0x7f6f4009a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:35.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.656+0000 7f6f45b66700 1 --2- 172.21.15.7:0/843940971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f40097530 0x7f6f4009a490 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400c510 tx=0x7f6f34007ea0 comp rx=0 tx=0).stop 2023-09-30T01:55:35.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.656+0000 7f6f45b66700 1 -- 172.21.15.7:0/843940971 shutdown_connections 2023-09-30T01:55:35.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.656+0000 7f6f45b66700 1 --2- 172.21.15.7:0/843940971 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6f2c0357e0 0x7f6f2c037c90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:35.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.656+0000 7f6f45b66700 1 --2- 172.21.15.7:0/843940971 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f400a2390 0x7f6f4009a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:35.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.656+0000 7f6f45b66700 1 --2- 172.21.15.7:0/843940971 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f40096a50 0x7f6f400a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:35.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.656+0000 7f6f45b66700 1 --2- 172.21.15.7:0/843940971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f40097530 0x7f6f4009a490 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:35.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.656+0000 7f6f45b66700 1 -- 172.21.15.7:0/843940971 >> 172.21.15.7:0/843940971 conn(0x7f6f40070cf0 msgr2=0x7f6f4013a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:35.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.656+0000 7f6f45b66700 1 -- 172.21.15.7:0/843940971 shutdown_connections 2023-09-30T01:55:35.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:35.656+0000 7f6f45b66700 1 -- 172.21.15.7:0/843940971 wait complete. 2023-09-30T01:55:38.668 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:55:38.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.797+0000 7f6bd57ca700 10 monclient: get_monmap_and_config 2023-09-30T01:55:38.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.797+0000 7f6bd57ca700 10 monclient: build_initial_monmap 2023-09-30T01:55:38.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.797+0000 7f6bd57ca700 10 monclient: monmap: 2023-09-30T01:55:38.797 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:38.797 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:38.797 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:55:38.798148+0000 2023-09-30T01:55:38.797 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:55:38.798148+0000 2023-09-30T01:55:38.797 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:38.798 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:38.798 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:55:38.798 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:55:38.798 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:55:38.798 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:55:38.798 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:55:38.798 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:38.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.797+0000 7f6bd57ca700 1 Processor -- start 2023-09-30T01:55:38.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.797+0000 7f6bd57ca700 1 -- start start 2023-09-30T01:55:38.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.797+0000 7f6bd57ca700 10 monclient: init 2023-09-30T01:55:38.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bd57ca700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:38.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bd57ca700 10 monclient: _add_conns ranks=[1,0,2,3,4] 2023-09-30T01:55:38.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bd57ca700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6bd0096a50 addr v1:172.21.15.50:6789/0 2023-09-30T01:55:38.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bd57ca700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6bd00a2390 addr v1:172.21.15.7:6789/0 2023-09-30T01:55:38.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bd57ca700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bd0097530 0x7f6bd013a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:38.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bd57ca700 10 monclient(hunting): picked mon.noname-b con 0x7f6bd0097530 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:38.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bd57ca700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd00980e0 con 0x7f6bd00a2390 2023-09-30T01:55:38.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bd57ca700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd0132eb0 con 0x7f6bd0096a50 2023-09-30T01:55:38.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bd57ca700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:38.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bd57ca700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd00991a0 con 0x7f6bd0097530 2023-09-30T01:55:38.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bd57ca700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bd57ca700 10 monclient(hunting): authenticate will time out at 1227.863244s 2023-09-30T01:55:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bcf7fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bd0097530 0x7f6bd013a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bce7fc700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6bd00a2390 0x7f6bd0096e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45962/0 (socket says 172.21.15.7:45962) 2023-09-30T01:55:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bce7fc700 1 -- 172.21.15.7:0/964726483 learned_addr learned my addr 172.21.15.7:0/964726483 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:55:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bcf7fe700 10 monclient(hunting): get_auth_request con 0x7f6bd0097530 auth_method 0 2023-09-30T01:55:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bcf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bcf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bcf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bcf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bcf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:38.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bcf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:38.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bcf7fe700 10 monclient(hunting): handle_auth_done global_id 4373 payload 274 2023-09-30T01:55:38.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bcf7fe700 1 -- 172.21.15.7:0/964726483 >> v1:172.21.15.50:6789/0 conn(0x7f6bd0096a50 legacy=0x7f6bd01327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:38.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bcf7fe700 1 -- 172.21.15.7:0/964726483 >> v1:172.21.15.7:6789/0 conn(0x7f6bd00a2390 legacy=0x7f6bd0096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:38.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bcf7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:55:38.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bcf7fe700 1 monclient: found mon.noname-b 2023-09-30T01:55:38.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bcf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:38.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bcf7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:55:38.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.798+0000 7f6bcf7fe700 1 -- 172.21.15.7:0/964726483 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd0099480 con 0x7f6bd0097530 2023-09-30T01:55:38.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.800+0000 7f6bcf7fe700 1 -- 172.21.15.7:0/964726483 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bd0097530 msgr2=0x7f6bd013a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:38.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.800+0000 7f6bcf7fe700 1 --2- 172.21.15.7:0/964726483 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bd0097530 0x7f6bd013a730 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8009910 tx=0x7f6bc80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7902fd7f82cb9a1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:38.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bcdffb700 1 -- 172.21.15.7:0/964726483 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc800b9f0 con 0x7f6bd0097530 2023-09-30T01:55:38.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bcdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:38.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bcdffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:55:38.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bcdffb700 10 monclient: dump: 2023-09-30T01:55:38.809 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:38.809 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:38.809 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:38.809 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:38.809 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:38.809 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:38.809 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:38.809 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:38.809 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:38.809 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:38.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bcdffb700 10 monclient: _finish_auth 0 2023-09-30T01:55:38.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bcdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:38.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bd57ca700 5 monclient: authenticate success, global_id 4373 2023-09-30T01:55:38.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bcdffb700 1 -- 172.21.15.7:0/964726483 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc800bb50 con 0x7f6bd0097530 2023-09-30T01:55:38.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bd57ca700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:55:38.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bcdffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:38.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bd57ca700 10 monclient: get_monmap_and_config success 2023-09-30T01:55:38.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bd57ca700 4 set_mon_vals no callback set 2023-09-30T01:55:38.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bcdffb700 1 -- 172.21.15.7:0/964726483 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8011420 con 0x7f6bd0097530 2023-09-30T01:55:38.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bcdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:38.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bcdffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:55:38.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bcdffb700 10 monclient: dump: 2023-09-30T01:55:38.811 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:38.811 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:38.811 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:38.811 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:38.811 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:38.811 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:38.811 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:38.811 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:38.811 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:38.811 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:38.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bd57ca700 10 monclient: shutdown 2023-09-30T01:55:38.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bd57ca700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:38.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bd57ca700 1 -- 172.21.15.7:0/964726483 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bd0097530 msgr2=0x7f6bd013a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:38.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bd57ca700 1 --2- 172.21.15.7:0/964726483 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bd0097530 0x7f6bd013a730 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8009910 tx=0x7f6bc80091a0 comp rx=0 tx=0).stop 2023-09-30T01:55:38.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bd57ca700 1 -- 172.21.15.7:0/964726483 shutdown_connections 2023-09-30T01:55:38.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bd57ca700 1 --2- 172.21.15.7:0/964726483 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bd0097530 0x7f6bd013a730 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:38.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bd57ca700 1 -- 172.21.15.7:0/964726483 >> 172.21.15.7:0/964726483 conn(0x7f6bd0070cf0 msgr2=0x7f6bd00710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:38.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bd57ca700 1 -- 172.21.15.7:0/964726483 shutdown_connections 2023-09-30T01:55:38.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.801+0000 7f6bd57ca700 1 -- 172.21.15.7:0/964726483 wait complete. 2023-09-30T01:55:38.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.802+0000 7f6bd57ca700 10 monclient: build_initial_monmap 2023-09-30T01:55:38.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.802+0000 7f6bd57ca700 10 monclient: monmap: 2023-09-30T01:55:38.813 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:38.813 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:55:38.813 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:55:38.813 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:55:38.813 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:38.813 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:38.813 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:55:38.813 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:55:38.813 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:55:38.813 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:38.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.802+0000 7f6bd57ca700 1 Processor -- start 2023-09-30T01:55:38.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.802+0000 7f6bd57ca700 1 -- start start 2023-09-30T01:55:38.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.802+0000 7f6bd57ca700 10 monclient: init 2023-09-30T01:55:38.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bd57ca700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:38.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bd57ca700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:55:38.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bd57ca700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bd0096a50 0x7f6bd009a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:38.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bd57ca700 10 monclient(hunting): picked mon.noname-c con 0x7f6bd0096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:38.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bd57ca700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd0097530 0x7f6bd009abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:38.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bd57ca700 10 monclient(hunting): picked mon.noname-a con 0x7f6bd0097530 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:55:38.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bd57ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bd00a2390 0x7f6bd009b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:38.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bd57ca700 10 monclient(hunting): picked mon.noname-b con 0x7f6bd00a2390 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:55:38.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bd57ca700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:38.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bd57ca700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd0133550 con 0x7f6bd0097530 2023-09-30T01:55:38.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bd57ca700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:38.814 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bd57ca700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd009e0d0 con 0x7f6bd00a2390 2023-09-30T01:55:38.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bd57ca700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:38.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bd57ca700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd009c3c0 con 0x7f6bd0096a50 2023-09-30T01:55:38.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bd57ca700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:38.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bd57ca700 10 monclient(hunting): authenticate will time out at 1227.868363s 2023-09-30T01:55:38.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bce7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd0097530 0x7f6bd009abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:38.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bceffd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bd0096a50 0x7f6bd009a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:38.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bce7fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd0097530 0x7f6bd009abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55848/0 (socket says 172.21.15.7:55848) 2023-09-30T01:55:38.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bceffd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bd0096a50 0x7f6bd009a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:50624/0 (socket says 172.21.15.7:50624) 2023-09-30T01:55:38.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bce7fc700 1 -- 172.21.15.7:0/1038661207 learned_addr learned my addr 172.21.15.7:0/1038661207 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:38.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bce7fc700 10 monclient(hunting): get_auth_request con 0x7f6bd0097530 auth_method 0 2023-09-30T01:55:38.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bce7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:38.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bce7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:38.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bce7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:38.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bceffd700 10 monclient(hunting): get_auth_request con 0x7f6bd0096a50 auth_method 0 2023-09-30T01:55:38.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bceffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:38.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bceffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:38.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bceffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:38.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bce7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:38.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bce7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:38.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bce7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:38.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bceffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:38.816 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bceffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:38.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bceffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:38.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bce7fc700 10 monclient(hunting): handle_auth_done global_id 4374 payload 824 2023-09-30T01:55:38.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bcf7fe700 1 --2- 172.21.15.7:0/1038661207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bd00a2390 0x7f6bd009b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:38.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bce7fc700 1 -- 172.21.15.7:0/1038661207 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bd0096a50 msgr2=0x7f6bd009a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:38.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bce7fc700 1 --2- 172.21.15.7:0/1038661207 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bd0096a50 0x7f6bd009a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:38.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bce7fc700 1 -- 172.21.15.7:0/1038661207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bd00a2390 msgr2=0x7f6bd009b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:38.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bce7fc700 1 --2- 172.21.15.7:0/1038661207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bd00a2390 0x7f6bd009b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:38.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bce7fc700 10 monclient: _finish_hunting 0 2023-09-30T01:55:38.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bce7fc700 1 monclient: found mon.noname-a 2023-09-30T01:55:38.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bce7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:38.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bce7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:55:38.817 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bce7fc700 1 -- 172.21.15.7:0/1038661207 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc8009420 con 0x7f6bd0097530 2023-09-30T01:55:38.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.803+0000 7f6bceffd700 1 --2- 172.21.15.7:0/1038661207 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bd0096a50 0x7f6bd009a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:55:38.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bce7fc700 1 -- 172.21.15.7:0/1038661207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd0097530 msgr2=0x7f6bd009abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:38.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bce7fc700 1 --2- 172.21.15.7:0/1038661207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd0097530 0x7f6bd009abd0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6bb800c5e0 tx=0x7f6bb8004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:38.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bc6ffd700 1 -- 172.21.15.7:0/1038661207 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb8005d40 con 0x7f6bd0097530 2023-09-30T01:55:38.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bc6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:38.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bc6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:55:38.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bc6ffd700 10 monclient: dump: 2023-09-30T01:55:38.820 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:38.820 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:38.820 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:38.820 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:38.821 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:38.821 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:38.821 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:38.821 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:38.821 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:38.821 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:38.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bc6ffd700 10 monclient: _finish_auth 0 2023-09-30T01:55:38.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bc6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:38.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bd57ca700 5 monclient: authenticate success, global_id 4374 2023-09-30T01:55:38.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bc6ffd700 1 -- 172.21.15.7:0/1038661207 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb80070d0 con 0x7f6bd0097530 2023-09-30T01:55:38.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bd57ca700 10 monclient: _renew_subs 2023-09-30T01:55:38.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bd57ca700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:38.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bd57ca700 1 -- 172.21.15.7:0/1038661207 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd00a52a0 con 0x7f6bd0097530 2023-09-30T01:55:38.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bc6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:38.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bc6ffd700 1 -- 172.21.15.7:0/1038661207 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb8007490 con 0x7f6bd0097530 2023-09-30T01:55:38.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bc6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:38.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bc6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:55:38.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bcd7fa700 4 set_mon_vals no callback set 2023-09-30T01:55:38.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bc6ffd700 10 monclient: dump: 2023-09-30T01:55:38.822 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:38.822 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:38.822 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:38.822 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:38.822 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:38.823 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:38.823 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:38.823 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:38.823 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:38.823 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:38.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bd57ca700 10 monclient: _renew_subs 2023-09-30T01:55:38.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bd57ca700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:38.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bd57ca700 1 -- 172.21.15.7:0/1038661207 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd009cf00 con 0x7f6bd0097530 2023-09-30T01:55:38.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bd57ca700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:38.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bc6ffd700 1 -- 172.21.15.7:0/1038661207 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f6bb801b3b0 con 0x7f6bd0097530 2023-09-30T01:55:38.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bd57ca700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:38.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bd57ca700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:38.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bd57ca700 1 -- 172.21.15.7:0/1038661207 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd0097330 con 0x7f6bd0097530 2023-09-30T01:55:38.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bc6ffd700 1 --2- 172.21.15.7:0/1038661207 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6bb00357e0 0x7f6bb0037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:38.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bc6ffd700 1 -- 172.21.15.7:0/1038661207 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f6bb8047180 con 0x7f6bd0097530 2023-09-30T01:55:38.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bceffd700 1 --2- 172.21.15.7:0/1038661207 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6bb00357e0 0x7f6bb0037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:38.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.807+0000 7f6bceffd700 10 monclient: get_auth_request con 0x7f6bb00357e0 auth_method 0 2023-09-30T01:55:38.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.810+0000 7f6bceffd700 1 --2- 172.21.15.7:0/1038661207 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6bb00357e0 0x7f6bb0037c90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0007fd0 tx=0x7f6bc00026a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:38.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.811+0000 7f6bc6ffd700 1 -- 172.21.15.7:0/1038661207 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6bb800f670 con 0x7f6bd0097530 2023-09-30T01:55:38.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.812+0000 7f6bc6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:38.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:38.812+0000 7f6bc6ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:55:39.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:39.047+0000 7f6bd57ca700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:55:39.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:39.047+0000 7f6bd57ca700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:39.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:39.047+0000 7f6bd57ca700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:39.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:39.047+0000 7f6bd57ca700 1 -- 172.21.15.7:0/1038661207 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6bd009d0f0 con 0x7f6bd0097530 2023-09-30T01:55:39.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:39.047+0000 7f6bc6ffd700 1 -- 172.21.15.7:0/1038661207 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6bb800f260 con 0x7f6bd0097530 2023-09-30T01:55:39.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:39.047+0000 7f6bc6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:39.048 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:39.047+0000 7f6bc6ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:55:39.048 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:55:39.048 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:55:39.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:39.049+0000 7f6bd57ca700 1 -- 172.21.15.7:0/1038661207 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6bb00357e0 msgr2=0x7f6bb0037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:39.049 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:39.049+0000 7f6bd57ca700 1 --2- 172.21.15.7:0/1038661207 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6bb00357e0 0x7f6bb0037c90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0007fd0 tx=0x7f6bc00026a0 comp rx=0 tx=0).stop 2023-09-30T01:55:39.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:39.049+0000 7f6bd57ca700 10 monclient: shutdown 2023-09-30T01:55:39.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:39.049+0000 7f6bd57ca700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:39.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:39.049+0000 7f6bd57ca700 1 -- 172.21.15.7:0/1038661207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd0097530 msgr2=0x7f6bd009abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:39.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:39.049+0000 7f6bd57ca700 1 --2- 172.21.15.7:0/1038661207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd0097530 0x7f6bd009abd0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6bb800c5e0 tx=0x7f6bb8004a80 comp rx=0 tx=0).stop 2023-09-30T01:55:39.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:39.049+0000 7f6bd57ca700 1 -- 172.21.15.7:0/1038661207 shutdown_connections 2023-09-30T01:55:39.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:39.049+0000 7f6bd57ca700 1 --2- 172.21.15.7:0/1038661207 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6bb00357e0 0x7f6bb0037c90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:39.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:39.049+0000 7f6bd57ca700 1 --2- 172.21.15.7:0/1038661207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bd00a2390 0x7f6bd009b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:39.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:39.049+0000 7f6bd57ca700 1 --2- 172.21.15.7:0/1038661207 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bd0096a50 0x7f6bd009a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:39.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:39.049+0000 7f6bd57ca700 1 --2- 172.21.15.7:0/1038661207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bd0097530 0x7f6bd009abd0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:39.050 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:39.049+0000 7f6bd57ca700 1 -- 172.21.15.7:0/1038661207 >> 172.21.15.7:0/1038661207 conn(0x7f6bd0070cf0 msgr2=0x7f6bd0131340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:39.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:39.049+0000 7f6bd57ca700 1 -- 172.21.15.7:0/1038661207 shutdown_connections 2023-09-30T01:55:39.051 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:39.049+0000 7f6bd57ca700 1 -- 172.21.15.7:0/1038661207 wait complete. 2023-09-30T01:55:42.062 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:55:42.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.189+0000 7f6222fe6700 10 monclient: get_monmap_and_config 2023-09-30T01:55:42.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.189+0000 7f6222fe6700 10 monclient: build_initial_monmap 2023-09-30T01:55:42.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.189+0000 7f6222fe6700 10 monclient: monmap: 2023-09-30T01:55:42.189 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:42.189 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:42.189 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:55:42.190241+0000 2023-09-30T01:55:42.190 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:55:42.190241+0000 2023-09-30T01:55:42.190 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:42.190 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:42.190 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:55:42.190 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:55:42.190 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:55:42.190 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:55:42.190 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:55:42.190 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:42.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.189+0000 7f6222fe6700 1 Processor -- start 2023-09-30T01:55:42.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.189+0000 7f6222fe6700 1 -- start start 2023-09-30T01:55:42.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.189+0000 7f6222fe6700 10 monclient: init 2023-09-30T01:55:42.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6222fe6700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:42.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6222fe6700 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-09-30T01:55:42.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6222fe6700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f621c096a50 0x7f621c132680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:42.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6222fe6700 10 monclient(hunting): picked mon.noname-b con 0x7f621c096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:42.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6222fe6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f621c096e20 0x7f621c0971f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:42.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6222fe6700 10 monclient(hunting): picked mon.noname-a con 0x7f621c096e20 addr v2:172.21.15.7:3300/0 2023-09-30T01:55:42.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6222fe6700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f621c12ef00 0x7f621c12f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:42.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6222fe6700 10 monclient(hunting): picked mon.noname-c con 0x7f621c12ef00 addr v2:172.21.15.50:3300/0 2023-09-30T01:55:42.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6222fe6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:42.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6222fe6700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f621c099440 con 0x7f621c096e20 2023-09-30T01:55:42.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6222fe6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:42.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6222fe6700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f621c097b40 con 0x7f621c12ef00 2023-09-30T01:55:42.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6222fe6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:42.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6222fe6700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f621c099160 con 0x7f621c096a50 2023-09-30T01:55:42.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6222fe6700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:42.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6222fe6700 10 monclient(hunting): authenticate will time out at 1231.255418s 2023-09-30T01:55:42.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6220d82700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f621c096a50 0x7f621c132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:42.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f621bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f621c096e20 0x7f621c0971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:42.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6220d82700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f621c096a50 0x7f621c132680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:50634/0 (socket says 172.21.15.7:50634) 2023-09-30T01:55:42.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f621bfff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f621c096e20 0x7f621c0971f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55852/0 (socket says 172.21.15.7:55852) 2023-09-30T01:55:42.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6220d82700 1 -- 172.21.15.7:0/990432419 learned_addr learned my addr 172.21.15.7:0/990432419 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:42.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6220d82700 10 monclient(hunting): get_auth_request con 0x7f621c096a50 auth_method 0 2023-09-30T01:55:42.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6220d82700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:42.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6220d82700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:42.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6220d82700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:42.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f621bfff700 10 monclient(hunting): get_auth_request con 0x7f621c096e20 auth_method 0 2023-09-30T01:55:42.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f621bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:42.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f621bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:42.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f621bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:42.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6220d82700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:42.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6220d82700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:42.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6220d82700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:42.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6221583700 1 --2- 172.21.15.7:0/990432419 >> v2:172.21.15.50:3300/0 conn(0x7f621c12ef00 0x7f621c12f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:42.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f621bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:42.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f621bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:42.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f621bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:42.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6221583700 10 monclient(hunting): get_auth_request con 0x7f621c12ef00 auth_method 0 2023-09-30T01:55:42.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6221583700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:42.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6221583700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:42.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6221583700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:42.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6220d82700 10 monclient(hunting): handle_auth_done global_id 4379 payload 274 2023-09-30T01:55:42.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6220d82700 1 -- 172.21.15.7:0/990432419 >> v2:172.21.15.50:3300/0 conn(0x7f621c12ef00 msgr2=0x7f621c12f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:42.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6220d82700 1 --2- 172.21.15.7:0/990432419 >> v2:172.21.15.50:3300/0 conn(0x7f621c12ef00 0x7f621c12f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:42.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6220d82700 1 -- 172.21.15.7:0/990432419 >> v2:172.21.15.7:3300/0 conn(0x7f621c096e20 msgr2=0x7f621c0971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:42.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6220d82700 1 --2- 172.21.15.7:0/990432419 >> v2:172.21.15.7:3300/0 conn(0x7f621c096e20 0x7f621c0971f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:42.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6220d82700 10 monclient: _finish_hunting 0 2023-09-30T01:55:42.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6220d82700 1 monclient: found mon.noname-b 2023-09-30T01:55:42.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6220d82700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:42.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6220d82700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:55:42.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6220d82700 1 -- 172.21.15.7:0/990432419 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f621c099720 con 0x7f621c096a50 2023-09-30T01:55:42.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f6221583700 1 --2- 172.21.15.7:0/990432419 >> v2:172.21.15.50:3300/0 conn(0x7f621c12ef00 0x7f621c12f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:55:42.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.190+0000 7f621bfff700 1 --2- 172.21.15.7:0/990432419 >> v2:172.21.15.7:3300/0 conn(0x7f621c096e20 0x7f621c0971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:55:42.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f6220d82700 1 -- 172.21.15.7:0/990432419 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f621c096a50 msgr2=0x7f621c132680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:42.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f6220d82700 1 --2- 172.21.15.7:0/990432419 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f621c096a50 0x7f621c132680 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f621c0a5710 tx=0x7f6210009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2abeb04c6f56cac2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:42.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f621b7fe700 1 -- 172.21.15.7:0/990432419 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6210009e70 con 0x7f621c096a50 2023-09-30T01:55:42.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f621b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:42.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f621b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:55:42.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f621b7fe700 10 monclient: dump: 2023-09-30T01:55:42.203 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:42.203 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:42.203 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:42.203 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:42.203 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:42.203 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:42.203 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:42.203 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:42.203 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:42.203 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:42.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f621b7fe700 10 monclient: _finish_auth 0 2023-09-30T01:55:42.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f621b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:42.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f6222fe6700 5 monclient: authenticate success, global_id 4379 2023-09-30T01:55:42.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f621b7fe700 1 -- 172.21.15.7:0/990432419 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621000ba70 con 0x7f621c096a50 2023-09-30T01:55:42.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f621b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:42.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f621b7fe700 1 -- 172.21.15.7:0/990432419 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621000be30 con 0x7f621c096a50 2023-09-30T01:55:42.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f621b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:42.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f621b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:55:42.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f621b7fe700 10 monclient: dump: 2023-09-30T01:55:42.204 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:42.204 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:42.204 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:42.204 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:42.205 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:42.205 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:42.205 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:42.205 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:42.205 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:42.205 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:42.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f6222fe6700 10 monclient: get_monmap_and_config success 2023-09-30T01:55:42.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f6222fe6700 4 set_mon_vals no callback set 2023-09-30T01:55:42.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f6222fe6700 10 monclient: shutdown 2023-09-30T01:55:42.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f6222fe6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:42.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f6222fe6700 1 -- 172.21.15.7:0/990432419 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f621c096a50 msgr2=0x7f621c132680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:42.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f6222fe6700 1 --2- 172.21.15.7:0/990432419 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f621c096a50 0x7f621c132680 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f621c0a5710 tx=0x7f6210009250 comp rx=0 tx=0).stop 2023-09-30T01:55:42.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f6222fe6700 1 -- 172.21.15.7:0/990432419 shutdown_connections 2023-09-30T01:55:42.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f6222fe6700 1 --2- 172.21.15.7:0/990432419 >> v2:172.21.15.50:3300/0 conn(0x7f621c12ef00 0x7f621c12f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:42.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f6222fe6700 1 --2- 172.21.15.7:0/990432419 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f621c096a50 0x7f621c132680 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:42.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f6222fe6700 1 --2- 172.21.15.7:0/990432419 >> v2:172.21.15.7:3300/0 conn(0x7f621c096e20 0x7f621c0971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:42.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f6222fe6700 1 -- 172.21.15.7:0/990432419 >> 172.21.15.7:0/990432419 conn(0x7f621c070cf0 msgr2=0x7f621c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:42.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.193+0000 7f6222fe6700 1 -- 172.21.15.7:0/990432419 shutdown_connections 2023-09-30T01:55:42.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.194+0000 7f6222fe6700 1 -- 172.21.15.7:0/990432419 wait complete. 2023-09-30T01:55:42.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.194+0000 7f6222fe6700 10 monclient: build_initial_monmap 2023-09-30T01:55:42.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.194+0000 7f6222fe6700 10 monclient: monmap: 2023-09-30T01:55:42.206 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:42.206 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:55:42.206 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:55:42.207 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:55:42.207 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:42.207 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:42.207 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:55:42.207 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:55:42.207 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:55:42.207 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:42.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.194+0000 7f6222fe6700 1 Processor -- start 2023-09-30T01:55:42.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6222fe6700 1 -- start start 2023-09-30T01:55:42.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6222fe6700 10 monclient: init 2023-09-30T01:55:42.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6222fe6700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:42.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6222fe6700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:55:42.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6222fe6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f621c096a50 0x7f621c0a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:42.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6222fe6700 10 monclient(hunting): picked mon.noname-b con 0x7f621c096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:55:42.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6222fe6700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f621c096e20 0x7f621c1339b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:42.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6222fe6700 10 monclient(hunting): picked mon.noname-c con 0x7f621c096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:42.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6222fe6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f621c12ef00 0x7f621c12dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:42.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6222fe6700 10 monclient(hunting): picked mon.noname-a con 0x7f621c12ef00 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:55:42.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6222fe6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:42.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6222fe6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621c133ef0 con 0x7f621c12ef00 2023-09-30T01:55:42.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6222fe6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:42.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6222fe6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621c11e3f0 con 0x7f621c096a50 2023-09-30T01:55:42.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6222fe6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:42.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6222fe6700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f621c142630 con 0x7f621c096e20 2023-09-30T01:55:42.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6222fe6700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:42.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6222fe6700 10 monclient(hunting): authenticate will time out at 1231.260565s 2023-09-30T01:55:42.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6221583700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f621c12ef00 0x7f621c12dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:42.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f621bfff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f621c096e20 0x7f621c1339b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:42.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6221583700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f621c12ef00 0x7f621c12dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55860/0 (socket says 172.21.15.7:55860) 2023-09-30T01:55:42.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6221583700 1 -- 172.21.15.7:0/1105627511 learned_addr learned my addr 172.21.15.7:0/1105627511 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:42.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f621bfff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f621c096e20 0x7f621c1339b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:50644/0 (socket says 172.21.15.7:50644) 2023-09-30T01:55:42.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6221583700 10 monclient(hunting): get_auth_request con 0x7f621c12ef00 auth_method 0 2023-09-30T01:55:42.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6221583700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:42.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6221583700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:42.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6221583700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:42.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f621bfff700 10 monclient(hunting): get_auth_request con 0x7f621c096e20 auth_method 0 2023-09-30T01:55:42.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f621bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:42.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f621bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:42.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f621bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:42.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6221583700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:42.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6221583700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:42.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6221583700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:42.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f621bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:42.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f621bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:42.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f621bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:42.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6220d82700 1 --2- 172.21.15.7:0/1105627511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f621c096a50 0x7f621c0a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:42.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6220d82700 10 monclient(hunting): get_auth_request con 0x7f621c096a50 auth_method 0 2023-09-30T01:55:42.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6220d82700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:42.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6220d82700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:42.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6220d82700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:42.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6221583700 10 monclient(hunting): handle_auth_done global_id 4380 payload 824 2023-09-30T01:55:42.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.195+0000 7f6221583700 1 -- 172.21.15.7:0/1105627511 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f621c096e20 msgr2=0x7f621c1339b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:42.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.196+0000 7f6221583700 1 --2- 172.21.15.7:0/1105627511 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f621c096e20 0x7f621c1339b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:42.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.196+0000 7f6221583700 1 -- 172.21.15.7:0/1105627511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f621c096a50 msgr2=0x7f621c0a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:42.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.196+0000 7f6221583700 1 --2- 172.21.15.7:0/1105627511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f621c096a50 0x7f621c0a0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:42.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.196+0000 7f6221583700 10 monclient: _finish_hunting 0 2023-09-30T01:55:42.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.196+0000 7f6221583700 1 monclient: found mon.noname-a 2023-09-30T01:55:42.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.196+0000 7f6221583700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:42.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.196+0000 7f6221583700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:55:42.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.196+0000 7f6221583700 1 -- 172.21.15.7:0/1105627511 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6208009420 con 0x7f621c12ef00 2023-09-30T01:55:42.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.196+0000 7f621bfff700 1 --2- 172.21.15.7:0/1105627511 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f621c096e20 0x7f621c1339b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:55:42.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f6221583700 1 -- 172.21.15.7:0/1105627511 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f621c12ef00 msgr2=0x7f621c12dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:42.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f6221583700 1 --2- 172.21.15.7:0/1105627511 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f621c12ef00 0x7f621c12dfc0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f621400d180 tx=0x7f6214009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:42.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f6218ff9700 1 -- 172.21.15.7:0/1105627511 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6214015440 con 0x7f621c12ef00 2023-09-30T01:55:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f6218ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f6218ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:55:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f6218ff9700 10 monclient: dump: 2023-09-30T01:55:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:42.216 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f6218ff9700 10 monclient: _finish_auth 0 2023-09-30T01:55:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f6218ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f6218ff9700 1 -- 172.21.15.7:0/1105627511 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621400f040 con 0x7f621c12ef00 2023-09-30T01:55:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f6222fe6700 5 monclient: authenticate success, global_id 4380 2023-09-30T01:55:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f6222fe6700 10 monclient: _renew_subs 2023-09-30T01:55:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f6222fe6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f6222fe6700 1 -- 172.21.15.7:0/1105627511 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62100094f0 con 0x7f621c12ef00 2023-09-30T01:55:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f6218ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f6218ff9700 1 -- 172.21.15.7:0/1105627511 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62140157f0 con 0x7f621c12ef00 2023-09-30T01:55:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f6218ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f6218ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:55:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f6218ff9700 10 monclient: dump: 2023-09-30T01:55:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:42.217 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:42.218 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f6222fe6700 10 monclient: _renew_subs 2023-09-30T01:55:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f621affd700 4 set_mon_vals no callback set 2023-09-30T01:55:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f6222fe6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.198+0000 7f6222fe6700 1 -- 172.21.15.7:0/1105627511 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f621c0a1710 con 0x7f621c12ef00 2023-09-30T01:55:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.199+0000 7f6218ff9700 1 -- 172.21.15.7:0/1105627511 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f621401e3b0 con 0x7f621c12ef00 2023-09-30T01:55:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.199+0000 7f6222fe6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.199+0000 7f6222fe6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.199+0000 7f6222fe6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:42.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.199+0000 7f6222fe6700 1 -- 172.21.15.7:0/1105627511 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f621c11e250 con 0x7f621c12ef00 2023-09-30T01:55:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.199+0000 7f6218ff9700 1 --2- 172.21.15.7:0/1105627511 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f62000357e0 0x7f6200037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.199+0000 7f6218ff9700 1 -- 172.21.15.7:0/1105627511 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f62140485e0 con 0x7f621c12ef00 2023-09-30T01:55:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.199+0000 7f6220d82700 1 --2- 172.21.15.7:0/1105627511 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f62000357e0 0x7f6200037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.199+0000 7f6220d82700 10 monclient: get_auth_request con 0x7f62000357e0 auth_method 0 2023-09-30T01:55:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.202+0000 7f6220d82700 1 --2- 172.21.15.7:0/1105627511 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f62000357e0 0x7f6200037c90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f621c0a5710 tx=0x7f6210017000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.204+0000 7f6218ff9700 1 -- 172.21.15.7:0/1105627511 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f621c11e250 con 0x7f621c12ef00 2023-09-30T01:55:42.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.204+0000 7f6218ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:42.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.204+0000 7f6218ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:55:42.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.441+0000 7f6222fe6700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:55:42.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.441+0000 7f6222fe6700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:42.441 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.441+0000 7f6222fe6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:42.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.441+0000 7f6222fe6700 1 -- 172.21.15.7:0/1105627511 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f621c11d2a0 con 0x7f621c12ef00 2023-09-30T01:55:42.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.441+0000 7f6218ff9700 1 -- 172.21.15.7:0/1105627511 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f621401a100 con 0x7f621c12ef00 2023-09-30T01:55:42.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.441+0000 7f6218ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:42.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.441+0000 7f6218ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:55:42.442 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:55:42.442 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:55:42.443 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.443+0000 7f6222fe6700 1 -- 172.21.15.7:0/1105627511 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f62000357e0 msgr2=0x7f6200037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:42.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.443+0000 7f6222fe6700 1 --2- 172.21.15.7:0/1105627511 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f62000357e0 0x7f6200037c90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f621c0a5710 tx=0x7f6210017000 comp rx=0 tx=0).stop 2023-09-30T01:55:42.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.443+0000 7f6222fe6700 10 monclient: shutdown 2023-09-30T01:55:42.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.443+0000 7f6222fe6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:42.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.443+0000 7f6222fe6700 1 -- 172.21.15.7:0/1105627511 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f621c12ef00 msgr2=0x7f621c12dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:42.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.443+0000 7f6222fe6700 1 --2- 172.21.15.7:0/1105627511 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f621c12ef00 0x7f621c12dfc0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f621400d180 tx=0x7f6214009a50 comp rx=0 tx=0).stop 2023-09-30T01:55:42.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.443+0000 7f6222fe6700 1 -- 172.21.15.7:0/1105627511 shutdown_connections 2023-09-30T01:55:42.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.443+0000 7f6222fe6700 1 --2- 172.21.15.7:0/1105627511 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f62000357e0 0x7f6200037c90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:42.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.443+0000 7f6222fe6700 1 --2- 172.21.15.7:0/1105627511 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f621c12ef00 0x7f621c12dfc0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:42.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.443+0000 7f6222fe6700 1 --2- 172.21.15.7:0/1105627511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f621c096a50 0x7f621c0a0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:42.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.443+0000 7f6222fe6700 1 --2- 172.21.15.7:0/1105627511 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f621c096e20 0x7f621c1339b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:42.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.443+0000 7f6222fe6700 1 -- 172.21.15.7:0/1105627511 >> 172.21.15.7:0/1105627511 conn(0x7f621c070cf0 msgr2=0x7f621c130ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:42.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.443+0000 7f6222fe6700 1 -- 172.21.15.7:0/1105627511 shutdown_connections 2023-09-30T01:55:42.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:42.443+0000 7f6222fe6700 1 -- 172.21.15.7:0/1105627511 wait complete. 2023-09-30T01:55:45.456 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:55:45.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.585+0000 7f7079c5e700 10 monclient: get_monmap_and_config 2023-09-30T01:55:45.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.585+0000 7f7079c5e700 10 monclient: build_initial_monmap 2023-09-30T01:55:45.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.585+0000 7f7079c5e700 10 monclient: monmap: 2023-09-30T01:55:45.586 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:45.586 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:45.586 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:55:45.586767+0000 2023-09-30T01:55:45.586 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:55:45.586767+0000 2023-09-30T01:55:45.586 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:45.586 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:45.586 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:55:45.586 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:55:45.587 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:55:45.587 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:55:45.587 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:55:45.587 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:45.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.586+0000 7f7079c5e700 1 Processor -- start 2023-09-30T01:55:45.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.586+0000 7f7079c5e700 1 -- start start 2023-09-30T01:55:45.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.586+0000 7f7079c5e700 10 monclient: init 2023-09-30T01:55:45.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.586+0000 7f7079c5e700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:45.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.586+0000 7f7079c5e700 10 monclient: _add_conns ranks=[3,2,1,0,4] 2023-09-30T01:55:45.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.586+0000 7f7079c5e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7074098540 0x7f70741329e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:45.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.586+0000 7f7079c5e700 10 monclient(hunting): picked mon.noname-a con 0x7f7074098540 addr v2:172.21.15.7:3300/0 2023-09-30T01:55:45.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.586+0000 7f7079c5e700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7074098910 0x7f7074098ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:45.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.586+0000 7f7079c5e700 10 monclient(hunting): picked mon.noname-b con 0x7f7074098910 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:45.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7079c5e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f707412ebc0 addr v1:172.21.15.50:6789/0 2023-09-30T01:55:45.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7079c5e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7074069f10 con 0x7f707412ebc0 2023-09-30T01:55:45.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7079c5e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:45.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7079c5e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70740a0f40 con 0x7f7074098540 2023-09-30T01:55:45.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7079c5e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:45.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7079c5e700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7074132f20 con 0x7f7074098910 2023-09-30T01:55:45.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7079c5e700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:45.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7079c5e700 10 monclient(hunting): authenticate will time out at 1234.652044s 2023-09-30T01:55:45.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7078c5c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7074098540 0x7f70741329e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:45.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7073fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7074098910 0x7f7074098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:45.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7078c5c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7074098540 0x7f70741329e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54174/0 (socket says 172.21.15.7:54174) 2023-09-30T01:55:45.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7073fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7074098910 0x7f7074098ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:46428/0 (socket says 172.21.15.7:46428) 2023-09-30T01:55:45.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7078c5c700 1 -- 172.21.15.7:0/2822910395 learned_addr learned my addr 172.21.15.7:0/2822910395 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:45.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7078c5c700 10 monclient(hunting): get_auth_request con 0x7f7074098540 auth_method 0 2023-09-30T01:55:45.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7078c5c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:45.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7078c5c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:45.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7078c5c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:45.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7073fff700 10 monclient(hunting): get_auth_request con 0x7f7074098910 auth_method 0 2023-09-30T01:55:45.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7073fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:45.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7073fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:45.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7073fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:45.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7078c5c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:45.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7078c5c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:45.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7078c5c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:45.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7073fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:45.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7073fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:45.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7073fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:45.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7073fff700 10 monclient(hunting): handle_auth_done global_id 4385 payload 274 2023-09-30T01:55:45.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7073fff700 1 -- 172.21.15.7:0/2822910395 >> v2:172.21.15.7:3300/0 conn(0x7f7074098540 msgr2=0x7f70741329e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:45.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7073fff700 1 --2- 172.21.15.7:0/2822910395 >> v2:172.21.15.7:3300/0 conn(0x7f7074098540 0x7f70741329e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:45.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7073fff700 1 -- 172.21.15.7:0/2822910395 >> v1:172.21.15.50:6789/0 conn(0x7f707412ebc0 legacy=0x7f707412f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:45.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7073fff700 10 monclient: _finish_hunting 0 2023-09-30T01:55:45.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7073fff700 1 monclient: found mon.noname-b 2023-09-30T01:55:45.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7073fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:45.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7073fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:55:45.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7073fff700 1 -- 172.21.15.7:0/2822910395 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7074096a50 con 0x7f7074098910 2023-09-30T01:55:45.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.587+0000 7f7078c5c700 1 --2- 172.21.15.7:0/2822910395 >> v2:172.21.15.7:3300/0 conn(0x7f7074098540 0x7f70741329e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:55:45.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.589+0000 7f7073fff700 1 -- 172.21.15.7:0/2822910395 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7074098910 msgr2=0x7f7074098ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:45.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.589+0000 7f7073fff700 1 --2- 172.21.15.7:0/2822910395 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7074098910 0x7f7074098ce0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f705c0098e0 tx=0x7f705c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11af4c38d0ad8745 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:45.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.589+0000 7f70737fe700 1 -- 172.21.15.7:0/2822910395 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c011840 con 0x7f7074098910 2023-09-30T01:55:45.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.589+0000 7f70737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:45.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.589+0000 7f70737fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:55:45.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.589+0000 7f70737fe700 10 monclient: dump: 2023-09-30T01:55:45.596 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:45.596 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:45.596 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:45.596 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:45.596 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:45.596 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:45.596 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:45.596 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:45.597 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:45.597 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:45.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.589+0000 7f70737fe700 10 monclient: _finish_auth 0 2023-09-30T01:55:45.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.589+0000 7f70737fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:45.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.589+0000 7f70737fe700 1 -- 172.21.15.7:0/2822910395 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705c0119a0 con 0x7f7074098910 2023-09-30T01:55:45.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.589+0000 7f7079c5e700 5 monclient: authenticate success, global_id 4385 2023-09-30T01:55:45.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.589+0000 7f7079c5e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:55:45.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.589+0000 7f70737fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:45.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.590+0000 7f7079c5e700 10 monclient: get_monmap_and_config success 2023-09-30T01:55:45.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.589+0000 7f70737fe700 1 -- 172.21.15.7:0/2822910395 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c011d40 con 0x7f7074098910 2023-09-30T01:55:45.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.590+0000 7f7079c5e700 4 set_mon_vals no callback set 2023-09-30T01:55:45.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.590+0000 7f70737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:45.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.590+0000 7f70737fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:55:45.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.590+0000 7f70737fe700 10 monclient: dump: 2023-09-30T01:55:45.598 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:45.598 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:45.598 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:45.598 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:45.598 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:45.598 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:45.598 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:45.598 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:45.599 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:45.599 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:45.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.590+0000 7f7079c5e700 10 monclient: shutdown 2023-09-30T01:55:45.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.590+0000 7f7079c5e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:45.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.590+0000 7f7079c5e700 1 -- 172.21.15.7:0/2822910395 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7074098910 msgr2=0x7f7074098ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:45.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.590+0000 7f7079c5e700 1 --2- 172.21.15.7:0/2822910395 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7074098910 0x7f7074098ce0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f705c0098e0 tx=0x7f705c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:55:45.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.590+0000 7f7079c5e700 1 -- 172.21.15.7:0/2822910395 shutdown_connections 2023-09-30T01:55:45.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.590+0000 7f7079c5e700 1 --2- 172.21.15.7:0/2822910395 >> v2:172.21.15.7:3300/0 conn(0x7f7074098540 0x7f70741329e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:45.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.590+0000 7f7079c5e700 1 --2- 172.21.15.7:0/2822910395 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7074098910 0x7f7074098ce0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:45.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.590+0000 7f7079c5e700 1 -- 172.21.15.7:0/2822910395 >> 172.21.15.7:0/2822910395 conn(0x7f7074071830 msgr2=0x7f70740a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:45.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.590+0000 7f7079c5e700 1 -- 172.21.15.7:0/2822910395 shutdown_connections 2023-09-30T01:55:45.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.590+0000 7f7079c5e700 1 -- 172.21.15.7:0/2822910395 wait complete. 2023-09-30T01:55:45.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.591+0000 7f7079c5e700 10 monclient: build_initial_monmap 2023-09-30T01:55:45.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.591+0000 7f7079c5e700 10 monclient: monmap: 2023-09-30T01:55:45.600 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:45.600 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:55:45.600 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:55:45.600 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:55:45.600 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:45.600 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:45.600 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:55:45.600 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:55:45.600 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:55:45.600 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:45.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.591+0000 7f7079c5e700 1 Processor -- start 2023-09-30T01:55:45.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.591+0000 7f7079c5e700 1 -- start start 2023-09-30T01:55:45.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.591+0000 7f7079c5e700 10 monclient: init 2023-09-30T01:55:45.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.591+0000 7f7079c5e700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7079c5e700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:55:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7079c5e700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7074098540 0x7f707409ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7079c5e700 10 monclient(hunting): picked mon.noname-c con 0x7f7074098540 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7079c5e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7074098910 0x7f707411ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7079c5e700 10 monclient(hunting): picked mon.noname-a con 0x7f7074098910 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:55:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7079c5e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f707412ebc0 0x7f707411bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7079c5e700 10 monclient(hunting): picked mon.noname-b con 0x7f707412ebc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:55:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7079c5e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7079c5e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707406b180 con 0x7f7074098910 2023-09-30T01:55:45.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7079c5e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7079c5e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707411ecb0 con 0x7f707412ebc0 2023-09-30T01:55:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7079c5e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7079c5e700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f707411e9d0 con 0x7f7074098540 2023-09-30T01:55:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7079c5e700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7079c5e700 10 monclient(hunting): authenticate will time out at 1234.657098s 2023-09-30T01:55:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7078c5c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7074098540 0x7f707409ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7073fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7074098910 0x7f707411ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7078c5c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7074098540 0x7f707409ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:46438/0 (socket says 172.21.15.7:46438) 2023-09-30T01:55:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7078c5c700 1 -- 172.21.15.7:0/2212923639 learned_addr learned my addr 172.21.15.7:0/2212923639 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:45.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7078c5c700 10 monclient(hunting): get_auth_request con 0x7f7074098540 auth_method 0 2023-09-30T01:55:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7078c5c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7078c5c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7078c5c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7073fff700 10 monclient(hunting): get_auth_request con 0x7f7074098910 auth_method 0 2023-09-30T01:55:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7073fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7073fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7073fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7078c5c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7078c5c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7078c5c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:45.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7073fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7073fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7073fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f707945d700 1 --2- 172.21.15.7:0/2212923639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f707412ebc0 0x7f707411bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7073fff700 10 monclient(hunting): handle_auth_done global_id 4386 payload 824 2023-09-30T01:55:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7073fff700 1 -- 172.21.15.7:0/2212923639 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7074098540 msgr2=0x7f707409ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7073fff700 1 --2- 172.21.15.7:0/2212923639 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7074098540 0x7f707409ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7073fff700 1 -- 172.21.15.7:0/2212923639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f707412ebc0 msgr2=0x7f707411bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7073fff700 1 --2- 172.21.15.7:0/2212923639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f707412ebc0 0x7f707411bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7073fff700 10 monclient: _finish_hunting 0 2023-09-30T01:55:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7073fff700 1 monclient: found mon.noname-a 2023-09-30T01:55:45.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7073fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:45.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7073fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:55:45.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7073fff700 1 -- 172.21.15.7:0/2212923639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7064009ae0 con 0x7f7074098910 2023-09-30T01:55:45.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f707945d700 10 monclient: get_auth_request con 0x7f707412ebc0 auth_method 0 2023-09-30T01:55:45.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f7078c5c700 1 --2- 172.21.15.7:0/2212923639 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7074098540 0x7f707409ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:55:45.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.592+0000 7f707945d700 1 --2- 172.21.15.7:0/2212923639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f707412ebc0 0x7f707411bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:55:45.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.594+0000 7f7073fff700 1 -- 172.21.15.7:0/2212923639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7074098910 msgr2=0x7f707411ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:45.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.594+0000 7f7073fff700 1 --2- 172.21.15.7:0/2212923639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7074098910 0x7f707411ba20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f707409f9d0 tx=0x7f705c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:45.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.594+0000 7f7070ff9700 1 -- 172.21.15.7:0/2212923639 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c00bbc0 con 0x7f7074098910 2023-09-30T01:55:45.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.594+0000 7f7070ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:45.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.594+0000 7f7070ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:55:45.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.594+0000 7f7070ff9700 10 monclient: dump: 2023-09-30T01:55:45.611 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:45.612 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:45.612 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:45.612 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:45.612 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:45.612 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:45.612 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:45.612 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:45.612 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:45.612 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:45.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.594+0000 7f7070ff9700 10 monclient: _finish_auth 0 2023-09-30T01:55:45.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.594+0000 7f7070ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:45.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.594+0000 7f7070ff9700 1 -- 172.21.15.7:0/2212923639 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705c020070 con 0x7f7074098910 2023-09-30T01:55:45.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.594+0000 7f7079c5e700 5 monclient: authenticate success, global_id 4386 2023-09-30T01:55:45.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.594+0000 7f7079c5e700 10 monclient: _renew_subs 2023-09-30T01:55:45.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.594+0000 7f7079c5e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:45.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.594+0000 7f7079c5e700 1 -- 172.21.15.7:0/2212923639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f705c009420 con 0x7f7074098910 2023-09-30T01:55:45.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.594+0000 7f7070ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:45.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.595+0000 7f7070ff9700 1 -- 172.21.15.7:0/2212923639 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705c01b450 con 0x7f7074098910 2023-09-30T01:55:45.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.595+0000 7f7070ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:45.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.595+0000 7f7070ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:55:45.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.595+0000 7f7070ff9700 10 monclient: dump: 2023-09-30T01:55:45.614 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:45.614 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:45.614 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:45.614 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:45.614 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:45.614 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:45.614 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:45.614 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:45.614 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:45.614 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:45.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.595+0000 7f7072ffd700 4 set_mon_vals no callback set 2023-09-30T01:55:45.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.595+0000 7f7079c5e700 10 monclient: _renew_subs 2023-09-30T01:55:45.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.595+0000 7f7079c5e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:45.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.595+0000 7f7079c5e700 1 -- 172.21.15.7:0/2212923639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70740a05c0 con 0x7f7074098910 2023-09-30T01:55:45.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.595+0000 7f7070ff9700 1 -- 172.21.15.7:0/2212923639 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f705c022030 con 0x7f7074098910 2023-09-30T01:55:45.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.595+0000 7f7070ff9700 1 --2- 172.21.15.7:0/2212923639 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f70540357e0 0x7f7054037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:45.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.596+0000 7f7070ff9700 1 -- 172.21.15.7:0/2212923639 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f705c049870 con 0x7f7074098910 2023-09-30T01:55:45.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.596+0000 7f7079c5e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:45.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.596+0000 7f7079c5e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:45.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.596+0000 7f7079c5e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:45.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.596+0000 7f7079c5e700 1 -- 172.21.15.7:0/2212923639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f707412f3f0 con 0x7f7074098910 2023-09-30T01:55:45.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.596+0000 7f7078c5c700 1 --2- 172.21.15.7:0/2212923639 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f70540357e0 0x7f7054037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:45.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.596+0000 7f7078c5c700 10 monclient: get_auth_request con 0x7f70540357e0 auth_method 0 2023-09-30T01:55:45.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.598+0000 7f7078c5c700 1 --2- 172.21.15.7:0/2212923639 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f70540357e0 0x7f7054037c90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f706400eb00 tx=0x7f706400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:45.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.600+0000 7f7070ff9700 1 -- 172.21.15.7:0/2212923639 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f705c0249c0 con 0x7f7074098910 2023-09-30T01:55:45.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.600+0000 7f7070ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:45.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.600+0000 7f7070ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:55:45.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.837+0000 7f7079c5e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:55:45.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.837+0000 7f7079c5e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:45.837 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.837+0000 7f7079c5e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:45.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.837+0000 7f7079c5e700 1 -- 172.21.15.7:0/2212923639 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f707412f3f0 con 0x7f7074098910 2023-09-30T01:55:45.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.837+0000 7f7070ff9700 1 -- 172.21.15.7:0/2212923639 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f705c0193f0 con 0x7f7074098910 2023-09-30T01:55:45.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.837+0000 7f7070ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:45.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.837+0000 7f7070ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:55:45.838 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:55:45.838 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:55:45.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.839+0000 7f7079c5e700 1 -- 172.21.15.7:0/2212923639 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f70540357e0 msgr2=0x7f7054037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:45.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.839+0000 7f7079c5e700 1 --2- 172.21.15.7:0/2212923639 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f70540357e0 0x7f7054037c90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f706400eb00 tx=0x7f706400b040 comp rx=0 tx=0).stop 2023-09-30T01:55:45.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.839+0000 7f7079c5e700 10 monclient: shutdown 2023-09-30T01:55:45.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.839+0000 7f7079c5e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:45.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.839+0000 7f7079c5e700 1 -- 172.21.15.7:0/2212923639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7074098910 msgr2=0x7f707411ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:45.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.839+0000 7f7079c5e700 1 --2- 172.21.15.7:0/2212923639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7074098910 0x7f707411ba20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f707409f9d0 tx=0x7f705c016000 comp rx=0 tx=0).stop 2023-09-30T01:55:45.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.839+0000 7f7079c5e700 1 -- 172.21.15.7:0/2212923639 shutdown_connections 2023-09-30T01:55:45.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.839+0000 7f7079c5e700 1 --2- 172.21.15.7:0/2212923639 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f70540357e0 0x7f7054037c90 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:45.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.839+0000 7f7079c5e700 1 --2- 172.21.15.7:0/2212923639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f707412ebc0 0x7f707411bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:45.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.839+0000 7f7079c5e700 1 --2- 172.21.15.7:0/2212923639 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7074098540 0x7f707409ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:45.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.839+0000 7f7079c5e700 1 --2- 172.21.15.7:0/2212923639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7074098910 0x7f707411ba20 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:45.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.839+0000 7f7079c5e700 1 -- 172.21.15.7:0/2212923639 >> 172.21.15.7:0/2212923639 conn(0x7f7074071830 msgr2=0x7f7074131880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:45.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.839+0000 7f7079c5e700 1 -- 172.21.15.7:0/2212923639 shutdown_connections 2023-09-30T01:55:45.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:45.839+0000 7f7079c5e700 1 -- 172.21.15.7:0/2212923639 wait complete. 2023-09-30T01:55:48.852 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:55:48.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.981+0000 7f7b13c0e700 10 monclient: get_monmap_and_config 2023-09-30T01:55:48.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.981+0000 7f7b13c0e700 10 monclient: build_initial_monmap 2023-09-30T01:55:48.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.981+0000 7f7b13c0e700 10 monclient: monmap: 2023-09-30T01:55:48.982 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:48.982 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:48.982 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:55:48.982714+0000 2023-09-30T01:55:48.982 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:55:48.982714+0000 2023-09-30T01:55:48.982 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:48.982 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:48.982 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:55:48.982 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:55:48.982 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:55:48.982 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:55:48.983 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:55:48.983 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:48.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.982+0000 7f7b13c0e700 1 Processor -- start 2023-09-30T01:55:48.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.982+0000 7f7b13c0e700 1 -- start start 2023-09-30T01:55:48.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.982+0000 7f7b13c0e700 10 monclient: init 2023-09-30T01:55:48.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.982+0000 7f7b13c0e700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:48.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.982+0000 7f7b13c0e700 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-09-30T01:55:48.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.982+0000 7f7b13c0e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b0c096a50 addr v1:172.21.15.7:6789/0 2023-09-30T01:55:48.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.982+0000 7f7b13c0e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7b0c0a2390 0x7f7b0c096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:48.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.982+0000 7f7b13c0e700 10 monclient(hunting): picked mon.noname-a con 0x7f7b0c0a2390 addr v2:172.21.15.7:3300/0 2023-09-30T01:55:48.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.982+0000 7f7b13c0e700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7b0c097360 0x7f7b0c139720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:48.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.982+0000 7f7b13c0e700 10 monclient(hunting): picked mon.noname-b con 0x7f7b0c097360 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:48.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.982+0000 7f7b13c0e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b0c0980e0 con 0x7f7b0c096a50 2023-09-30T01:55:48.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.982+0000 7f7b13c0e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:48.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.982+0000 7f7b13c0e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b0c132eb0 con 0x7f7b0c0a2390 2023-09-30T01:55:48.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.982+0000 7f7b13c0e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:48.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.982+0000 7f7b13c0e700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c0991a0 con 0x7f7b0c097360 2023-09-30T01:55:48.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.982+0000 7f7b13c0e700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:48.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.982+0000 7f7b13c0e700 10 monclient(hunting): authenticate will time out at 1238.047971s 2023-09-30T01:55:48.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b111a9700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f7b0c0a2390 0x7f7b0c096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:48.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b121ab700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7b0c097360 0x7f7b0c139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:48.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b119aa700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7b0c096a50 0x7f7b0c1327a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59346/0 (socket says 172.21.15.7:59346) 2023-09-30T01:55:48.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b119aa700 1 -- 172.21.15.7:0/164406416 learned_addr learned my addr 172.21.15.7:0/164406416 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:55:48.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b111a9700 10 monclient(hunting): get_auth_request con 0x7f7b0c0a2390 auth_method 0 2023-09-30T01:55:48.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b111a9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:48.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b111a9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:48.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b111a9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:48.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b121ab700 10 monclient(hunting): get_auth_request con 0x7f7b0c097360 auth_method 0 2023-09-30T01:55:48.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b121ab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:48.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b121ab700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:48.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b121ab700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:48.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b111a9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:48.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b111a9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:48.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b111a9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:48.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b121ab700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:48.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b121ab700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:48.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b121ab700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:48.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b111a9700 10 monclient(hunting): handle_auth_done global_id 4389 payload 274 2023-09-30T01:55:48.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b111a9700 1 -- 172.21.15.7:0/164406416 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7b0c097360 msgr2=0x7f7b0c139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:48.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b111a9700 1 --2- 172.21.15.7:0/164406416 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7b0c097360 0x7f7b0c139720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:48.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b111a9700 1 -- 172.21.15.7:0/164406416 >> v1:172.21.15.7:6789/0 conn(0x7f7b0c096a50 legacy=0x7f7b0c1327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:48.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b111a9700 10 monclient: _finish_hunting 0 2023-09-30T01:55:48.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b111a9700 1 monclient: found mon.noname-a 2023-09-30T01:55:48.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b111a9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:48.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b111a9700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:55:48.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b111a9700 1 -- 172.21.15.7:0/164406416 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b0c099480 con 0x7f7b0c0a2390 2023-09-30T01:55:48.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.983+0000 7f7b121ab700 1 --2- 172.21.15.7:0/164406416 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7b0c097360 0x7f7b0c139720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:55:48.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.985+0000 7f7b111a9700 1 -- 172.21.15.7:0/164406416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b0c0a2390 msgr2=0x7f7b0c096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:48.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.985+0000 7f7b111a9700 1 --2- 172.21.15.7:0/164406416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b0c0a2390 0x7f7b0c096e20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0098e0 tx=0x7f7afc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df2d9ce45a7bbcf7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:48.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.985+0000 7f7b109a8700 1 -- 172.21.15.7:0/164406416 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc011840 con 0x7f7b0c0a2390 2023-09-30T01:55:48.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.985+0000 7f7b109a8700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:48.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.985+0000 7f7b109a8700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:55:48.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.985+0000 7f7b109a8700 10 monclient: dump: 2023-09-30T01:55:48.994 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:48.994 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:48.994 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:48.994 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:48.994 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:48.994 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:48.994 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:48.994 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:48.994 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:48.994 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:48.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.985+0000 7f7b109a8700 10 monclient: _finish_auth 0 2023-09-30T01:55:48.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.985+0000 7f7b109a8700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:48.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.985+0000 7f7b13c0e700 5 monclient: authenticate success, global_id 4389 2023-09-30T01:55:48.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.985+0000 7f7b109a8700 1 -- 172.21.15.7:0/164406416 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7afc0119a0 con 0x7f7b0c0a2390 2023-09-30T01:55:48.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.985+0000 7f7b109a8700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:48.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.985+0000 7f7b13c0e700 10 monclient: get_monmap_and_config success 2023-09-30T01:55:48.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.985+0000 7f7b13c0e700 4 set_mon_vals no callback set 2023-09-30T01:55:48.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.985+0000 7f7b109a8700 1 -- 172.21.15.7:0/164406416 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc011d40 con 0x7f7b0c0a2390 2023-09-30T01:55:48.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.985+0000 7f7b109a8700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:48.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.985+0000 7f7b109a8700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:55:48.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.985+0000 7f7b109a8700 10 monclient: dump: 2023-09-30T01:55:48.996 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:48.996 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:48.996 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:48.996 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:48.996 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:48.996 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:48.996 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:48.996 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:48.996 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:48.996 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:48.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.986+0000 7f7b13c0e700 10 monclient: shutdown 2023-09-30T01:55:48.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.986+0000 7f7b13c0e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:48.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.986+0000 7f7b13c0e700 1 -- 172.21.15.7:0/164406416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b0c0a2390 msgr2=0x7f7b0c096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:48.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.986+0000 7f7b13c0e700 1 --2- 172.21.15.7:0/164406416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b0c0a2390 0x7f7b0c096e20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0098e0 tx=0x7f7afc0091a0 comp rx=0 tx=0).stop 2023-09-30T01:55:48.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.986+0000 7f7b13c0e700 1 -- 172.21.15.7:0/164406416 shutdown_connections 2023-09-30T01:55:48.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.986+0000 7f7b13c0e700 1 --2- 172.21.15.7:0/164406416 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7b0c097360 0x7f7b0c139720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:48.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.986+0000 7f7b13c0e700 1 --2- 172.21.15.7:0/164406416 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b0c0a2390 0x7f7b0c096e20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:48.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.986+0000 7f7b13c0e700 1 -- 172.21.15.7:0/164406416 >> 172.21.15.7:0/164406416 conn(0x7f7b0c070cf0 msgr2=0x7f7b0c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:48.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.986+0000 7f7b13c0e700 1 -- 172.21.15.7:0/164406416 shutdown_connections 2023-09-30T01:55:48.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.986+0000 7f7b13c0e700 1 -- 172.21.15.7:0/164406416 wait complete. 2023-09-30T01:55:48.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.986+0000 7f7b13c0e700 10 monclient: build_initial_monmap 2023-09-30T01:55:48.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.986+0000 7f7b13c0e700 10 monclient: monmap: 2023-09-30T01:55:48.997 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:48.997 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:55:48.998 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:55:48.998 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:55:48.998 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:48.998 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:48.998 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:55:48.998 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:55:48.998 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:55:48.998 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:49.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b13c0e700 1 Processor -- start 2023-09-30T01:55:49.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b13c0e700 1 -- start start 2023-09-30T01:55:49.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b13c0e700 10 monclient: init 2023-09-30T01:55:49.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b13c0e700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:49.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b13c0e700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:55:49.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b13c0e700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7b0c096a50 0x7f7b0c09ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:49.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b13c0e700 10 monclient(hunting): picked mon.noname-c con 0x7f7b0c096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:49.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b13c0e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b0c097360 0x7f7b0c09cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:49.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b13c0e700 10 monclient(hunting): picked mon.noname-b con 0x7f7b0c097360 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:55:49.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b13c0e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b0c0a2390 0x7f7b0c09d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:49.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b13c0e700 10 monclient(hunting): picked mon.noname-a con 0x7f7b0c0a2390 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:55:49.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b13c0e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:49.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b13c0e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c1334e0 con 0x7f7b0c0a2390 2023-09-30T01:55:49.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b13c0e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:49.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b13c0e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c132380 con 0x7f7b0c097360 2023-09-30T01:55:49.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b13c0e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:49.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b13c0e700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c0a0190 con 0x7f7b0c096a50 2023-09-30T01:55:49.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b13c0e700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:49.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b13c0e700 10 monclient(hunting): authenticate will time out at 1238.052801s 2023-09-30T01:55:49.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b119aa700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7b0c096a50 0x7f7b0c09ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:49.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b121ab700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b0c0a2390 0x7f7b0c09d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:49.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b119aa700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7b0c096a50 0x7f7b0c09ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:46456/0 (socket says 172.21.15.7:46456) 2023-09-30T01:55:49.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b119aa700 1 -- 172.21.15.7:0/284196149 learned_addr learned my addr 172.21.15.7:0/284196149 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:49.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b119aa700 10 monclient(hunting): get_auth_request con 0x7f7b0c096a50 auth_method 0 2023-09-30T01:55:49.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b119aa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:49.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b119aa700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:49.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b119aa700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:49.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b121ab700 10 monclient(hunting): get_auth_request con 0x7f7b0c0a2390 auth_method 0 2023-09-30T01:55:49.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b121ab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:49.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b121ab700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:49.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.987+0000 7f7b121ab700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:49.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b119aa700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:49.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b119aa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:49.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b119aa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:49.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b121ab700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:49.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b121ab700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:49.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b121ab700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:49.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b111a9700 1 --2- 172.21.15.7:0/284196149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b0c097360 0x7f7b0c09cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b111a9700 10 monclient(hunting): get_auth_request con 0x7f7b0c097360 auth_method 0 2023-09-30T01:55:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b111a9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b111a9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b111a9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b119aa700 10 monclient(hunting): handle_auth_done global_id 4394 payload 824 2023-09-30T01:55:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b119aa700 1 -- 172.21.15.7:0/284196149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b0c097360 msgr2=0x7f7b0c09cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b119aa700 1 --2- 172.21.15.7:0/284196149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b0c097360 0x7f7b0c09cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b119aa700 1 -- 172.21.15.7:0/284196149 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b0c0a2390 msgr2=0x7f7b0c09d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b119aa700 1 --2- 172.21.15.7:0/284196149 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b0c0a2390 0x7f7b0c09d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b119aa700 10 monclient: _finish_hunting 0 2023-09-30T01:55:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b119aa700 1 monclient: found mon.noname-c 2023-09-30T01:55:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b119aa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:49.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b119aa700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:55:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b119aa700 1 -- 172.21.15.7:0/284196149 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b04009a20 con 0x7f7b0c096a50 2023-09-30T01:55:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.988+0000 7f7b121ab700 1 --2- 172.21.15.7:0/284196149 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b0c0a2390 0x7f7b0c09d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:55:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b119aa700 1 -- 172.21.15.7:0/284196149 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7b0c096a50 msgr2=0x7f7b0c09ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b119aa700 1 --2- 172.21.15.7:0/284196149 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7b0c096a50 0x7f7b0c09ca30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7b0800c5e0 tx=0x7f7b08004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b01ffb700 1 -- 172.21.15.7:0/284196149 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b08005d40 con 0x7f7b0c096a50 2023-09-30T01:55:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b01ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b01ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:55:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b01ffb700 10 monclient: dump: 2023-09-30T01:55:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:49.014 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:49.015 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b01ffb700 10 monclient: _finish_auth 0 2023-09-30T01:55:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b01ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b13c0e700 5 monclient: authenticate success, global_id 4394 2023-09-30T01:55:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b01ffb700 1 -- 172.21.15.7:0/284196149 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b080070d0 con 0x7f7b0c096a50 2023-09-30T01:55:49.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b01ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:49.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b01ffb700 1 -- 172.21.15.7:0/284196149 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b08007490 con 0x7f7b0c096a50 2023-09-30T01:55:49.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b13c0e700 10 monclient: _renew_subs 2023-09-30T01:55:49.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b13c0e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:49.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b13c0e700 1 -- 172.21.15.7:0/284196149 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7afc009420 con 0x7f7b0c096a50 2023-09-30T01:55:49.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b03fff700 4 set_mon_vals no callback set 2023-09-30T01:55:49.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b01ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:49.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b01ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:55:49.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b01ffb700 10 monclient: dump: 2023-09-30T01:55:49.016 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:49.016 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:49.016 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:49.017 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:49.017 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:49.017 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:49.017 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:49.017 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:49.017 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:49.017 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:49.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b13c0e700 10 monclient: _renew_subs 2023-09-30T01:55:49.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b13c0e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:49.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.990+0000 7f7b13c0e700 1 -- 172.21.15.7:0/284196149 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b0c09fe80 con 0x7f7b0c096a50 2023-09-30T01:55:49.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.991+0000 7f7b01ffb700 1 -- 172.21.15.7:0/284196149 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f7b0801b3b0 con 0x7f7b0c096a50 2023-09-30T01:55:49.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.991+0000 7f7b01ffb700 1 --2- 172.21.15.7:0/284196149 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7af00357e0 0x7f7af0037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:49.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.991+0000 7f7b01ffb700 1 -- 172.21.15.7:0/284196149 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f7b080472c0 con 0x7f7b0c096a50 2023-09-30T01:55:49.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.991+0000 7f7b13c0e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:49.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.991+0000 7f7b13c0e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:49.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.991+0000 7f7b13c0e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:49.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.991+0000 7f7b13c0e700 1 -- 172.21.15.7:0/284196149 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b0c0a4820 con 0x7f7b0c096a50 2023-09-30T01:55:49.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.992+0000 7f7b111a9700 1 --2- 172.21.15.7:0/284196149 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7af00357e0 0x7f7af0037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:49.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.992+0000 7f7b111a9700 10 monclient: get_auth_request con 0x7f7af00357e0 auth_method 0 2023-09-30T01:55:49.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.994+0000 7f7b111a9700 1 --2- 172.21.15.7:0/284196149 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7af00357e0 0x7f7af0037c90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0030c0 tx=0x7f7afc016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:49.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.996+0000 7f7b01ffb700 1 -- 172.21.15.7:0/284196149 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7b08045e60 con 0x7f7b0c096a50 2023-09-30T01:55:49.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.996+0000 7f7b01ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:49.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:48.996+0000 7f7b01ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:55:49.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:49.235+0000 7f7b13c0e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:55:49.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:49.235+0000 7f7b13c0e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:49.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:49.235+0000 7f7b13c0e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:49.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:49.235+0000 7f7b13c0e700 1 -- 172.21.15.7:0/284196149 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7b0c0a4b00 con 0x7f7b0c096a50 2023-09-30T01:55:49.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:49.236+0000 7f7b01ffb700 1 -- 172.21.15.7:0/284196149 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7b08017070 con 0x7f7b0c096a50 2023-09-30T01:55:49.236 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:55:49.236 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:55:49.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:49.236+0000 7f7b01ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:49.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:49.236+0000 7f7b01ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:55:49.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:49.238+0000 7f7b13c0e700 1 -- 172.21.15.7:0/284196149 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7af00357e0 msgr2=0x7f7af0037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:49.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:49.238+0000 7f7b13c0e700 1 --2- 172.21.15.7:0/284196149 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7af00357e0 0x7f7af0037c90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0030c0 tx=0x7f7afc016000 comp rx=0 tx=0).stop 2023-09-30T01:55:49.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:49.238+0000 7f7b13c0e700 10 monclient: shutdown 2023-09-30T01:55:49.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:49.238+0000 7f7b13c0e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:49.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:49.238+0000 7f7b13c0e700 1 -- 172.21.15.7:0/284196149 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7b0c096a50 msgr2=0x7f7b0c09ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:49.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:49.238+0000 7f7b13c0e700 1 --2- 172.21.15.7:0/284196149 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7b0c096a50 0x7f7b0c09ca30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7b0800c5e0 tx=0x7f7b08004a80 comp rx=0 tx=0).stop 2023-09-30T01:55:49.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:49.238+0000 7f7b13c0e700 1 -- 172.21.15.7:0/284196149 shutdown_connections 2023-09-30T01:55:49.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:49.238+0000 7f7b13c0e700 1 --2- 172.21.15.7:0/284196149 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7af00357e0 0x7f7af0037c90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:49.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:49.238+0000 7f7b13c0e700 1 --2- 172.21.15.7:0/284196149 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b0c0a2390 0x7f7b0c09d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:49.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:49.238+0000 7f7b13c0e700 1 --2- 172.21.15.7:0/284196149 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7b0c096a50 0x7f7b0c09ca30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:49.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:49.238+0000 7f7b13c0e700 1 --2- 172.21.15.7:0/284196149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b0c097360 0x7f7b0c09cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:49.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:49.238+0000 7f7b13c0e700 1 -- 172.21.15.7:0/284196149 >> 172.21.15.7:0/284196149 conn(0x7f7b0c070cf0 msgr2=0x7f7b0c1336f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:49.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:49.238+0000 7f7b13c0e700 1 -- 172.21.15.7:0/284196149 shutdown_connections 2023-09-30T01:55:49.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:49.238+0000 7f7b13c0e700 1 -- 172.21.15.7:0/284196149 wait complete. 2023-09-30T01:55:52.251 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:55:52.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.383+0000 7f0c62f24700 10 monclient: get_monmap_and_config 2023-09-30T01:55:52.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.383+0000 7f0c62f24700 10 monclient: build_initial_monmap 2023-09-30T01:55:52.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.383+0000 7f0c62f24700 10 monclient: monmap: 2023-09-30T01:55:52.383 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:52.384 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:52.384 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:55:52.384223+0000 2023-09-30T01:55:52.384 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:55:52.384223+0000 2023-09-30T01:55:52.384 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:52.384 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:52.384 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:55:52.384 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:55:52.384 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:55:52.384 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:55:52.385 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:55:52.385 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.383+0000 7f0c62f24700 1 Processor -- start 2023-09-30T01:55:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.383+0000 7f0c62f24700 1 -- start start 2023-09-30T01:55:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.383+0000 7f0c62f24700 10 monclient: init 2023-09-30T01:55:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c62f24700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c62f24700 10 monclient: _add_conns ranks=[1,2,4,3,0] 2023-09-30T01:55:52.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c62f24700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0c5c096a50 addr v1:172.21.15.50:6789/0 2023-09-30T01:55:52.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c62f24700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c5c096e20 0x7f0c5c0971f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:52.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c62f24700 10 monclient(hunting): picked mon.noname-b con 0x7f0c5c096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:52.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c62f24700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0c5c137710 0x7f0c5c139af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:52.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c62f24700 10 monclient(hunting): picked mon.noname-c con 0x7f0c5c137710 addr v2:172.21.15.50:3300/0 2023-09-30T01:55:52.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c62f24700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c5c132d90 con 0x7f0c5c096a50 2023-09-30T01:55:52.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c62f24700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:52.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c62f24700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c5c097b40 con 0x7f0c5c137710 2023-09-30T01:55:52.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c62f24700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:52.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c62f24700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5c099160 con 0x7f0c5c096e20 2023-09-30T01:55:52.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c62f24700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:52.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c62f24700 10 monclient(hunting): authenticate will time out at 1241.449460s 2023-09-30T01:55:52.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c5bfff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c5c096e20 0x7f0c5c0971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:52.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c5bfff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c5c096e20 0x7f0c5c0971f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:46460/0 (socket says 172.21.15.7:46460) 2023-09-30T01:55:52.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c5bfff700 1 -- 172.21.15.7:0/632784230 learned_addr learned my addr 172.21.15.7:0/632784230 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:52.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c5bfff700 10 monclient(hunting): get_auth_request con 0x7f0c5c096e20 auth_method 0 2023-09-30T01:55:52.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c5bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:52.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c5bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:52.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c5bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:52.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c5bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:52.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c5bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:52.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c5bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:52.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c614c1700 1 --2- 172.21.15.7:0/632784230 >> v2:172.21.15.50:3300/0 conn(0x7f0c5c137710 0x7f0c5c139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:52.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c614c1700 10 monclient(hunting): get_auth_request con 0x7f0c5c137710 auth_method 0 2023-09-30T01:55:52.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c614c1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:52.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c614c1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:52.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c614c1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:52.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c5bfff700 10 monclient(hunting): handle_auth_done global_id 4397 payload 274 2023-09-30T01:55:52.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c5bfff700 1 -- 172.21.15.7:0/632784230 >> v2:172.21.15.50:3300/0 conn(0x7f0c5c137710 msgr2=0x7f0c5c139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:52.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c5bfff700 1 --2- 172.21.15.7:0/632784230 >> v2:172.21.15.50:3300/0 conn(0x7f0c5c137710 0x7f0c5c139af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:52.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c5bfff700 1 -- 172.21.15.7:0/632784230 >> v1:172.21.15.50:6789/0 conn(0x7f0c5c096a50 legacy=0x7f0c5c132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:52.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c5bfff700 10 monclient: _finish_hunting 0 2023-09-30T01:55:52.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c5bfff700 1 monclient: found mon.noname-b 2023-09-30T01:55:52.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c5bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:52.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c5bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:55:52.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c5bfff700 1 -- 172.21.15.7:0/632784230 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c5c099720 con 0x7f0c5c096e20 2023-09-30T01:55:52.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.384+0000 7f0c614c1700 1 --2- 172.21.15.7:0/632784230 >> v2:172.21.15.50:3300/0 conn(0x7f0c5c137710 0x7f0c5c139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:55:52.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c5bfff700 1 -- 172.21.15.7:0/632784230 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c5c096e20 msgr2=0x7f0c5c0971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:52.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c5bfff700 1 --2- 172.21.15.7:0/632784230 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c5c096e20 0x7f0c5c0971f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0c500099b0 tx=0x7f0c50009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b02c31ed0a557768 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:52.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c5b7fe700 1 -- 172.21.15.7:0/632784230 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5000ba30 con 0x7f0c5c096e20 2023-09-30T01:55:52.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c5b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:52.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c5b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:55:52.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c5b7fe700 10 monclient: dump: 2023-09-30T01:55:52.395 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:52.396 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:52.396 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:52.396 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:52.396 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:52.396 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:52.396 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:52.396 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:52.396 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:52.396 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:52.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c5b7fe700 10 monclient: _finish_auth 0 2023-09-30T01:55:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c5b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c62f24700 5 monclient: authenticate success, global_id 4397 2023-09-30T01:55:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c62f24700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:55:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c5b7fe700 1 -- 172.21.15.7:0/632784230 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5000bb90 con 0x7f0c5c096e20 2023-09-30T01:55:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c5b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c62f24700 10 monclient: get_monmap_and_config success 2023-09-30T01:55:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c62f24700 4 set_mon_vals no callback set 2023-09-30T01:55:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c5b7fe700 1 -- 172.21.15.7:0/632784230 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c50011420 con 0x7f0c5c096e20 2023-09-30T01:55:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c5b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c5b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:55:52.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c5b7fe700 10 monclient: dump: 2023-09-30T01:55:52.398 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:52.398 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:52.398 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:52.398 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:52.398 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:52.398 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:52.398 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:52.398 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:52.398 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:52.398 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:52.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c62f24700 10 monclient: shutdown 2023-09-30T01:55:52.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c62f24700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:52.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c62f24700 1 -- 172.21.15.7:0/632784230 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c5c096e20 msgr2=0x7f0c5c0971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:52.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c62f24700 1 --2- 172.21.15.7:0/632784230 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c5c096e20 0x7f0c5c0971f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0c500099b0 tx=0x7f0c50009250 comp rx=0 tx=0).stop 2023-09-30T01:55:52.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c62f24700 1 -- 172.21.15.7:0/632784230 shutdown_connections 2023-09-30T01:55:52.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c62f24700 1 --2- 172.21.15.7:0/632784230 >> v2:172.21.15.50:3300/0 conn(0x7f0c5c137710 0x7f0c5c139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:52.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c62f24700 1 --2- 172.21.15.7:0/632784230 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c5c096e20 0x7f0c5c0971f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:52.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c62f24700 1 -- 172.21.15.7:0/632784230 >> 172.21.15.7:0/632784230 conn(0x7f0c5c070cf0 msgr2=0x7f0c5c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:52.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c62f24700 1 -- 172.21.15.7:0/632784230 shutdown_connections 2023-09-30T01:55:52.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.387+0000 7f0c62f24700 1 -- 172.21.15.7:0/632784230 wait complete. 2023-09-30T01:55:52.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.388+0000 7f0c62f24700 10 monclient: build_initial_monmap 2023-09-30T01:55:52.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.388+0000 7f0c62f24700 10 monclient: monmap: 2023-09-30T01:55:52.399 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:52.399 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:55:52.400 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:55:52.400 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:55:52.400 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:52.400 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:52.400 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:55:52.400 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:55:52.400 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:55:52.400 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:52.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.388+0000 7f0c62f24700 1 Processor -- start 2023-09-30T01:55:52.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.388+0000 7f0c62f24700 1 -- start start 2023-09-30T01:55:52.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.388+0000 7f0c62f24700 10 monclient: init 2023-09-30T01:55:52.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c62f24700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:52.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c62f24700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:55:52.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c62f24700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c5c096a50 0x7f0c5c142a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:52.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c62f24700 10 monclient(hunting): picked mon.noname-c con 0x7f0c5c096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:52.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c62f24700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c5c096e20 0x7f0c5c1421b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:52.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c62f24700 10 monclient(hunting): picked mon.noname-a con 0x7f0c5c096e20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:55:52.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c62f24700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c5c137710 0x7f0c5c12f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:52.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c62f24700 10 monclient(hunting): picked mon.noname-b con 0x7f0c5c137710 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:55:52.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c62f24700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:52.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c62f24700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5c0a0290 con 0x7f0c5c096e20 2023-09-30T01:55:52.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c62f24700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:52.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c62f24700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5c1334e0 con 0x7f0c5c137710 2023-09-30T01:55:52.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c62f24700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:52.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c62f24700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5c131d60 con 0x7f0c5c096a50 2023-09-30T01:55:52.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c62f24700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:52.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c62f24700 10 monclient(hunting): authenticate will time out at 1241.454451s 2023-09-30T01:55:52.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c60cc0700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c5c096a50 0x7f0c5c142a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:52.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c5bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c5c096e20 0x7f0c5c1421b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:52.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c5bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c5c096e20 0x7f0c5c1421b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54212/0 (socket says 172.21.15.7:54212) 2023-09-30T01:55:52.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c60cc0700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c5c096a50 0x7f0c5c142a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:46476/0 (socket says 172.21.15.7:46476) 2023-09-30T01:55:52.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c5bfff700 1 -- 172.21.15.7:0/3073587071 learned_addr learned my addr 172.21.15.7:0/3073587071 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:52.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c5bfff700 10 monclient(hunting): get_auth_request con 0x7f0c5c096e20 auth_method 0 2023-09-30T01:55:52.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c5bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:52.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c5bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:52.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c5bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:52.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c60cc0700 10 monclient(hunting): get_auth_request con 0x7f0c5c096a50 auth_method 0 2023-09-30T01:55:52.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c60cc0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:52.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c60cc0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:52.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c60cc0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:52.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c5bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:52.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c5bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:52.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c5bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:52.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c60cc0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:52.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c60cc0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:52.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c60cc0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:52.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c614c1700 1 --2- 172.21.15.7:0/3073587071 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c5c137710 0x7f0c5c12f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:52.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c5bfff700 10 monclient(hunting): handle_auth_done global_id 4395 payload 824 2023-09-30T01:55:52.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c5bfff700 1 -- 172.21.15.7:0/3073587071 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c5c096a50 msgr2=0x7f0c5c142a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:52.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c5bfff700 1 --2- 172.21.15.7:0/3073587071 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c5c096a50 0x7f0c5c142a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:52.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c5bfff700 1 -- 172.21.15.7:0/3073587071 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c5c137710 msgr2=0x7f0c5c12f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:52.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c5bfff700 1 --2- 172.21.15.7:0/3073587071 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c5c137710 0x7f0c5c12f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:52.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c5bfff700 10 monclient: _finish_hunting 0 2023-09-30T01:55:52.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c5bfff700 1 monclient: found mon.noname-a 2023-09-30T01:55:52.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c5bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:52.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c5bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:55:52.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c5bfff700 1 -- 172.21.15.7:0/3073587071 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c500094f0 con 0x7f0c5c096e20 2023-09-30T01:55:52.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c614c1700 10 monclient: get_auth_request con 0x7f0c5c137710 auth_method 0 2023-09-30T01:55:52.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c614c1700 1 --2- 172.21.15.7:0/3073587071 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c5c137710 0x7f0c5c12f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:55:52.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.389+0000 7f0c60cc0700 1 --2- 172.21.15.7:0/3073587071 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c5c096a50 0x7f0c5c142a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:55:52.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c5bfff700 1 -- 172.21.15.7:0/3073587071 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c5c096e20 msgr2=0x7f0c5c1421b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:52.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c5bfff700 1 --2- 172.21.15.7:0/3073587071 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c5c096e20 0x7f0c5c1421b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c0992a0 tx=0x7f0c50016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:52.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c58ff9700 1 -- 172.21.15.7:0/3073587071 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c50026480 con 0x7f0c5c096e20 2023-09-30T01:55:52.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c58ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:52.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c58ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:55:52.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c58ff9700 10 monclient: dump: 2023-09-30T01:55:52.408 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:52.408 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:52.409 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:52.409 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:52.409 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:52.409 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:52.409 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:52.409 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:52.409 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:52.409 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:52.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c58ff9700 10 monclient: _finish_auth 0 2023-09-30T01:55:52.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c58ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:52.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c58ff9700 1 -- 172.21.15.7:0/3073587071 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5001f070 con 0x7f0c5c096e20 2023-09-30T01:55:52.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c62f24700 5 monclient: authenticate success, global_id 4395 2023-09-30T01:55:52.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c62f24700 10 monclient: _renew_subs 2023-09-30T01:55:52.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c62f24700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:52.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c62f24700 1 -- 172.21.15.7:0/3073587071 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c5c0a1840 con 0x7f0c5c096e20 2023-09-30T01:55:52.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c58ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:52.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c58ff9700 1 -- 172.21.15.7:0/3073587071 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c50026850 con 0x7f0c5c096e20 2023-09-30T01:55:52.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c58ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:52.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c58ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:55:52.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c58ff9700 10 monclient: dump: 2023-09-30T01:55:52.411 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:52.411 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:52.411 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:52.411 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:52.411 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:52.411 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:52.411 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:52.411 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:52.412 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:52.412 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:52.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c5affd700 4 set_mon_vals no callback set 2023-09-30T01:55:52.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c62f24700 10 monclient: _renew_subs 2023-09-30T01:55:52.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c62f24700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:52.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.392+0000 7f0c62f24700 1 -- 172.21.15.7:0/3073587071 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c5c0a08e0 con 0x7f0c5c096e20 2023-09-30T01:55:52.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.393+0000 7f0c58ff9700 1 -- 172.21.15.7:0/3073587071 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f0c5001a440 con 0x7f0c5c096e20 2023-09-30T01:55:52.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.393+0000 7f0c58ff9700 1 --2- 172.21.15.7:0/3073587071 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f0c480357e0 0x7f0c48037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:52.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.393+0000 7f0c58ff9700 1 -- 172.21.15.7:0/3073587071 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f0c50021070 con 0x7f0c5c096e20 2023-09-30T01:55:52.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.393+0000 7f0c62f24700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:52.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.393+0000 7f0c62f24700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:52.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.393+0000 7f0c62f24700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:52.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.393+0000 7f0c62f24700 1 -- 172.21.15.7:0/3073587071 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c5c0a30a0 con 0x7f0c5c096e20 2023-09-30T01:55:52.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.393+0000 7f0c60cc0700 1 --2- 172.21.15.7:0/3073587071 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f0c480357e0 0x7f0c48037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:52.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.393+0000 7f0c60cc0700 10 monclient: get_auth_request con 0x7f0c480357e0 auth_method 0 2023-09-30T01:55:52.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.396+0000 7f0c60cc0700 1 --2- 172.21.15.7:0/3073587071 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f0c480357e0 0x7f0c48037c90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c002d70 tx=0x7f0c4c007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:52.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.398+0000 7f0c58ff9700 1 -- 172.21.15.7:0/3073587071 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0c5002b020 con 0x7f0c5c096e20 2023-09-30T01:55:52.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.398+0000 7f0c58ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:52.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.398+0000 7f0c58ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:55:52.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.633+0000 7f0c62f24700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:55:52.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.633+0000 7f0c62f24700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:52.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.633+0000 7f0c62f24700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:52.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.633+0000 7f0c62f24700 1 -- 172.21.15.7:0/3073587071 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0c5c0712c0 con 0x7f0c5c096e20 2023-09-30T01:55:52.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.633+0000 7f0c58ff9700 1 -- 172.21.15.7:0/3073587071 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0c50034080 con 0x7f0c5c096e20 2023-09-30T01:55:52.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.633+0000 7f0c58ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:52.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.633+0000 7f0c58ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:55:52.635 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:55:52.635 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:55:52.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.635+0000 7f0c62f24700 1 -- 172.21.15.7:0/3073587071 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f0c480357e0 msgr2=0x7f0c48037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:52.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.635+0000 7f0c62f24700 1 --2- 172.21.15.7:0/3073587071 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f0c480357e0 0x7f0c48037c90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c002d70 tx=0x7f0c4c007ea0 comp rx=0 tx=0).stop 2023-09-30T01:55:52.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.635+0000 7f0c62f24700 10 monclient: shutdown 2023-09-30T01:55:52.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.635+0000 7f0c62f24700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:52.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.635+0000 7f0c62f24700 1 -- 172.21.15.7:0/3073587071 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c5c096e20 msgr2=0x7f0c5c1421b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:52.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.635+0000 7f0c62f24700 1 --2- 172.21.15.7:0/3073587071 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c5c096e20 0x7f0c5c1421b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c0992a0 tx=0x7f0c50016000 comp rx=0 tx=0).stop 2023-09-30T01:55:52.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.635+0000 7f0c62f24700 1 -- 172.21.15.7:0/3073587071 shutdown_connections 2023-09-30T01:55:52.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.635+0000 7f0c62f24700 1 --2- 172.21.15.7:0/3073587071 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f0c480357e0 0x7f0c48037c90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:52.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.635+0000 7f0c62f24700 1 --2- 172.21.15.7:0/3073587071 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c5c137710 0x7f0c5c12f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:52.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.635+0000 7f0c62f24700 1 --2- 172.21.15.7:0/3073587071 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0c5c096a50 0x7f0c5c142a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:52.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.635+0000 7f0c62f24700 1 --2- 172.21.15.7:0/3073587071 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0c5c096e20 0x7f0c5c1421b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:52.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.635+0000 7f0c62f24700 1 -- 172.21.15.7:0/3073587071 >> 172.21.15.7:0/3073587071 conn(0x7f0c5c070cf0 msgr2=0x7f0c5c135e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:52.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.636+0000 7f0c62f24700 1 -- 172.21.15.7:0/3073587071 shutdown_connections 2023-09-30T01:55:52.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:52.636+0000 7f0c62f24700 1 -- 172.21.15.7:0/3073587071 wait complete. 2023-09-30T01:55:55.648 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:55:55.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.779+0000 7f0d40293700 10 monclient: get_monmap_and_config 2023-09-30T01:55:55.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.779+0000 7f0d40293700 10 monclient: build_initial_monmap 2023-09-30T01:55:55.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.779+0000 7f0d40293700 10 monclient: monmap: 2023-09-30T01:55:55.780 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:55.780 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:55.780 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:55:55.780583+0000 2023-09-30T01:55:55.780 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:55:55.780583+0000 2023-09-30T01:55:55.780 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:55.780 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:55.780 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:55:55.780 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:55:55.780 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:55:55.780 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:55:55.780 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:55:55.781 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:55.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.780+0000 7f0d40293700 1 Processor -- start 2023-09-30T01:55:55.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.780+0000 7f0d40293700 1 -- start start 2023-09-30T01:55:55.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.780+0000 7f0d40293700 10 monclient: init 2023-09-30T01:55:55.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.780+0000 7f0d40293700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:55.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.780+0000 7f0d40293700 10 monclient: _add_conns ranks=[4,3,2,0,1] 2023-09-30T01:55:55.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.780+0000 7f0d40293700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0d38096a50 0x7f0d38132750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:55.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.780+0000 7f0d40293700 10 monclient(hunting): picked mon.noname-c con 0x7f0d38096a50 addr v2:172.21.15.50:3300/0 2023-09-30T01:55:55.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.780+0000 7f0d40293700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0d38096e20 0x7f0d38097240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:55.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.780+0000 7f0d40293700 10 monclient(hunting): picked mon.noname-a con 0x7f0d38096e20 addr v2:172.21.15.7:3300/0 2023-09-30T01:55:55.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.780+0000 7f0d40293700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0d3812ef00 0x7f0d3812f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:55.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.780+0000 7f0d40293700 10 monclient(hunting): picked mon.noname-b con 0x7f0d3812ef00 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:55.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.780+0000 7f0d40293700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:55.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.780+0000 7f0d40293700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d38099480 con 0x7f0d38096e20 2023-09-30T01:55:55.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.780+0000 7f0d40293700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:55.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.780+0000 7f0d40293700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d38097bf0 con 0x7f0d38096a50 2023-09-30T01:55:55.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.780+0000 7f0d40293700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:55.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.780+0000 7f0d40293700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d380991a0 con 0x7f0d3812ef00 2023-09-30T01:55:55.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.780+0000 7f0d40293700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:55.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.780+0000 7f0d40293700 10 monclient(hunting): authenticate will time out at 1244.845804s 2023-09-30T01:55:55.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.780+0000 7f0d3e830700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0d3812ef00 0x7f0d3812f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:55.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.780+0000 7f0d3d82e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f0d38096e20 0x7f0d38097240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:55.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e830700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0d3812ef00 0x7f0d3812f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:49858/0 (socket says 172.21.15.7:49858) 2023-09-30T01:55:55.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e830700 1 -- 172.21.15.7:0/1063549981 learned_addr learned my addr 172.21.15.7:0/1063549981 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:55.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e830700 10 monclient(hunting): get_auth_request con 0x7f0d3812ef00 auth_method 0 2023-09-30T01:55:55.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e830700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:55.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e830700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:55.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e830700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3d82e700 10 monclient(hunting): get_auth_request con 0x7f0d38096e20 auth_method 0 2023-09-30T01:55:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3d82e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3d82e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3d82e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3d82e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3d82e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3d82e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e830700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:55.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e830700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e02f700 1 --2- 172.21.15.7:0/1063549981 >> v2:172.21.15.50:3300/0 conn(0x7f0d38096a50 0x7f0d38132750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e830700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e02f700 10 monclient(hunting): get_auth_request con 0x7f0d38096a50 auth_method 0 2023-09-30T01:55:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e02f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e02f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e02f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e830700 10 monclient(hunting): handle_auth_done global_id 4403 payload 274 2023-09-30T01:55:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e830700 1 -- 172.21.15.7:0/1063549981 >> v2:172.21.15.50:3300/0 conn(0x7f0d38096a50 msgr2=0x7f0d38132750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e830700 1 --2- 172.21.15.7:0/1063549981 >> v2:172.21.15.50:3300/0 conn(0x7f0d38096a50 0x7f0d38132750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:55.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e830700 1 -- 172.21.15.7:0/1063549981 >> v2:172.21.15.7:3300/0 conn(0x7f0d38096e20 msgr2=0x7f0d38097240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:55.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e830700 1 --2- 172.21.15.7:0/1063549981 >> v2:172.21.15.7:3300/0 conn(0x7f0d38096e20 0x7f0d38097240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:55.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e830700 10 monclient: _finish_hunting 0 2023-09-30T01:55:55.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e830700 1 monclient: found mon.noname-b 2023-09-30T01:55:55.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e830700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:55.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e830700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:55:55.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3e830700 1 -- 172.21.15.7:0/1063549981 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d38099760 con 0x7f0d3812ef00 2023-09-30T01:55:55.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.781+0000 7f0d3d82e700 1 --2- 172.21.15.7:0/1063549981 >> v2:172.21.15.7:3300/0 conn(0x7f0d38096e20 0x7f0d38097240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:55:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.783+0000 7f0d3e830700 1 -- 172.21.15.7:0/1063549981 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0d3812ef00 msgr2=0x7f0d3812f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.783+0000 7f0d3e830700 1 --2- 172.21.15.7:0/1063549981 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0d3812ef00 0x7f0d3812f3a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0d38098000 tx=0x7f0d30009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb018f3edac478d1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.783+0000 7f0d3d02d700 1 -- 172.21.15.7:0/1063549981 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3000e070 con 0x7f0d3812ef00 2023-09-30T01:55:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.783+0000 7f0d3d02d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.783+0000 7f0d3d02d700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:55:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.783+0000 7f0d3d02d700 10 monclient: dump: 2023-09-30T01:55:55.796 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:55.797 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:55.797 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:55.797 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:55.797 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:55.797 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:55.797 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:55.797 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:55.797 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:55.797 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:55.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.783+0000 7f0d3d02d700 10 monclient: _finish_auth 0 2023-09-30T01:55:55.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.783+0000 7f0d3d02d700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:55.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.783+0000 7f0d3d02d700 1 -- 172.21.15.7:0/1063549981 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d3000bc30 con 0x7f0d3812ef00 2023-09-30T01:55:55.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.783+0000 7f0d40293700 5 monclient: authenticate success, global_id 4403 2023-09-30T01:55:55.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.783+0000 7f0d40293700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:55:55.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.783+0000 7f0d3d02d700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:55.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.783+0000 7f0d3d02d700 1 -- 172.21.15.7:0/1063549981 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d300092e0 con 0x7f0d3812ef00 2023-09-30T01:55:55.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.783+0000 7f0d3d02d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:55.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.783+0000 7f0d3d02d700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:55:55.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.783+0000 7f0d3d02d700 10 monclient: dump: 2023-09-30T01:55:55.799 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:55.799 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:55.799 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:55.799 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:55.799 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:55.799 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:55.799 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:55.799 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:55.799 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:55.799 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:55.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.783+0000 7f0d40293700 10 monclient: get_monmap_and_config success 2023-09-30T01:55:55.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.783+0000 7f0d40293700 4 set_mon_vals no callback set 2023-09-30T01:55:55.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.784+0000 7f0d40293700 10 monclient: shutdown 2023-09-30T01:55:55.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.784+0000 7f0d40293700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:55.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.784+0000 7f0d40293700 1 -- 172.21.15.7:0/1063549981 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0d3812ef00 msgr2=0x7f0d3812f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:55.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.784+0000 7f0d40293700 1 --2- 172.21.15.7:0/1063549981 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0d3812ef00 0x7f0d3812f3a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0d38098000 tx=0x7f0d30009840 comp rx=0 tx=0).stop 2023-09-30T01:55:55.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.784+0000 7f0d40293700 1 -- 172.21.15.7:0/1063549981 shutdown_connections 2023-09-30T01:55:55.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.784+0000 7f0d40293700 1 --2- 172.21.15.7:0/1063549981 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0d3812ef00 0x7f0d3812f3a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:55.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.784+0000 7f0d40293700 1 --2- 172.21.15.7:0/1063549981 >> v2:172.21.15.50:3300/0 conn(0x7f0d38096a50 0x7f0d38132750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:55.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.784+0000 7f0d40293700 1 --2- 172.21.15.7:0/1063549981 >> v2:172.21.15.7:3300/0 conn(0x7f0d38096e20 0x7f0d38097240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.784+0000 7f0d40293700 1 -- 172.21.15.7:0/1063549981 >> 172.21.15.7:0/1063549981 conn(0x7f0d38070cf0 msgr2=0x7f0d380710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.784+0000 7f0d40293700 1 -- 172.21.15.7:0/1063549981 shutdown_connections 2023-09-30T01:55:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.784+0000 7f0d40293700 1 -- 172.21.15.7:0/1063549981 wait complete. 2023-09-30T01:55:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.784+0000 7f0d40293700 10 monclient: build_initial_monmap 2023-09-30T01:55:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.784+0000 7f0d40293700 10 monclient: monmap: 2023-09-30T01:55:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:55:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:55:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:55:55.801 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:55:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:55:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:55:55.802 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d40293700 1 Processor -- start 2023-09-30T01:55:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d40293700 1 -- start start 2023-09-30T01:55:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d40293700 10 monclient: init 2023-09-30T01:55:55.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d40293700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d40293700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:55:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d40293700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0d38096a50 0x7f0d380a0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d40293700 10 monclient(hunting): picked mon.noname-c con 0x7f0d38096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d40293700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d38096e20 0x7f0d381339d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d40293700 10 monclient(hunting): picked mon.noname-b con 0x7f0d38096e20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:55:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d40293700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d3812ef00 0x7f0d3812dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d40293700 10 monclient(hunting): picked mon.noname-a con 0x7f0d3812ef00 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:55:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d40293700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:55.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d40293700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d380a6390 con 0x7f0d3812ef00 2023-09-30T01:55:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d40293700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d40293700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3811e760 con 0x7f0d38096e20 2023-09-30T01:55:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d40293700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d40293700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3811e110 con 0x7f0d38096a50 2023-09-30T01:55:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d40293700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d40293700 10 monclient(hunting): authenticate will time out at 1244.850906s 2023-09-30T01:55:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d3e02f700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0d38096a50 0x7f0d380a0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d3e02f700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0d38096a50 0x7f0d380a0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:49874/0 (socket says 172.21.15.7:49874) 2023-09-30T01:55:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.785+0000 7f0d3e830700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d3812ef00 0x7f0d3812dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:55.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e02f700 1 -- 172.21.15.7:0/2713514324 learned_addr learned my addr 172.21.15.7:0/2713514324 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:55.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e02f700 10 monclient(hunting): get_auth_request con 0x7f0d38096a50 auth_method 0 2023-09-30T01:55:55.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e02f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:55.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e02f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:55.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e02f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:55.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e830700 10 monclient(hunting): get_auth_request con 0x7f0d3812ef00 auth_method 0 2023-09-30T01:55:55.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e830700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:55.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e830700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:55.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e830700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:55.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e02f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:55.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e02f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:55.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3d82e700 1 --2- 172.21.15.7:0/2713514324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d38096e20 0x7f0d381339d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:55.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e02f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:55.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e830700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:55.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e830700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:55.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e830700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:55.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3d82e700 10 monclient(hunting): get_auth_request con 0x7f0d38096e20 auth_method 0 2023-09-30T01:55:55.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3d82e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:55.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3d82e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:55.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3d82e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:55.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3d82e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:55.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3d82e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:55.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3d82e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:55.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e830700 10 monclient(hunting): handle_auth_done global_id 4401 payload 824 2023-09-30T01:55:55.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e830700 1 -- 172.21.15.7:0/2713514324 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0d38096a50 msgr2=0x7f0d380a0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:55.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e830700 1 --2- 172.21.15.7:0/2713514324 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0d38096a50 0x7f0d380a0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:55.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e830700 1 -- 172.21.15.7:0/2713514324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d38096e20 msgr2=0x7f0d381339d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:55.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e830700 1 --2- 172.21.15.7:0/2713514324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d38096e20 0x7f0d381339d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:55.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e830700 10 monclient: _finish_hunting 0 2023-09-30T01:55:55.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e830700 1 monclient: found mon.noname-a 2023-09-30T01:55:55.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e830700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:55.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e830700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:55:55.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e830700 1 -- 172.21.15.7:0/2713514324 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2c009420 con 0x7f0d3812ef00 2023-09-30T01:55:55.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.786+0000 7f0d3e02f700 1 --2- 172.21.15.7:0/2713514324 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0d38096a50 0x7f0d380a0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:55:55.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d3e830700 1 -- 172.21.15.7:0/2713514324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d3812ef00 msgr2=0x7f0d3812dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:55.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d3e830700 1 --2- 172.21.15.7:0/2713514324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d3812ef00 0x7f0d3812dfc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0d381331d0 tx=0x7f0d30018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:55.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d367fc700 1 -- 172.21.15.7:0/2713514324 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d30008bc0 con 0x7f0d3812ef00 2023-09-30T01:55:55.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:55.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d367fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:55:55.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d367fc700 10 monclient: dump: 2023-09-30T01:55:55.809 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:55.809 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:55.809 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:55.809 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:55.809 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:55.809 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:55.809 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:55.809 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:55.809 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:55.809 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:55.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d367fc700 10 monclient: _finish_auth 0 2023-09-30T01:55:55.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d367fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:55.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d40293700 5 monclient: authenticate success, global_id 4401 2023-09-30T01:55:55.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d367fc700 1 -- 172.21.15.7:0/2713514324 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d30008d20 con 0x7f0d3812ef00 2023-09-30T01:55:55.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d367fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:55.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d367fc700 1 -- 172.21.15.7:0/2713514324 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d30026690 con 0x7f0d3812ef00 2023-09-30T01:55:55.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d40293700 10 monclient: _renew_subs 2023-09-30T01:55:55.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d40293700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:55.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d40293700 1 -- 172.21.15.7:0/2713514324 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d30009ae0 con 0x7f0d3812ef00 2023-09-30T01:55:55.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d3c82c700 4 set_mon_vals no callback set 2023-09-30T01:55:55.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:55.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d367fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:55:55.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d367fc700 10 monclient: dump: 2023-09-30T01:55:55.811 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:55.811 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:55.811 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:55.811 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:55.811 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:55.811 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:55.812 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:55.812 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:55.812 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:55.812 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:55.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d40293700 10 monclient: _renew_subs 2023-09-30T01:55:55.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d40293700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:55.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.788+0000 7f0d40293700 1 -- 172.21.15.7:0/2713514324 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d3811e4a0 con 0x7f0d3812ef00 2023-09-30T01:55:55.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.789+0000 7f0d40293700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:55.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.789+0000 7f0d367fc700 1 -- 172.21.15.7:0/2713514324 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f0d30003040 con 0x7f0d3812ef00 2023-09-30T01:55:55.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.789+0000 7f0d40293700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:55.812 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.789+0000 7f0d40293700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:55.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.789+0000 7f0d40293700 1 -- 172.21.15.7:0/2713514324 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d38132470 con 0x7f0d3812ef00 2023-09-30T01:55:55.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.790+0000 7f0d367fc700 1 --2- 172.21.15.7:0/2713514324 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f0d240357e0 0x7f0d24037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:55.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.790+0000 7f0d367fc700 1 -- 172.21.15.7:0/2713514324 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f0d30049aa0 con 0x7f0d3812ef00 2023-09-30T01:55:55.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.790+0000 7f0d3e02f700 1 --2- 172.21.15.7:0/2713514324 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f0d240357e0 0x7f0d24037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:55.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.790+0000 7f0d3e02f700 10 monclient: get_auth_request con 0x7f0d240357e0 auth_method 0 2023-09-30T01:55:55.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.793+0000 7f0d3e02f700 1 --2- 172.21.15.7:0/2713514324 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f0d240357e0 0x7f0d24037c90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0d28006fd0 tx=0x7f0d28008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:55.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.794+0000 7f0d367fc700 1 -- 172.21.15.7:0/2713514324 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0d3002fe00 con 0x7f0d3812ef00 2023-09-30T01:55:55.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.794+0000 7f0d367fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:55.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:55.794+0000 7f0d367fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:55:56.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:56.028+0000 7f0d40293700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:55:56.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:56.028+0000 7f0d40293700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:56.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:56.028+0000 7f0d40293700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:55:56.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:56.028+0000 7f0d40293700 1 -- 172.21.15.7:0/2713514324 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0d38064c10 con 0x7f0d3812ef00 2023-09-30T01:55:56.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:56.028+0000 7f0d367fc700 1 -- 172.21.15.7:0/2713514324 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0d30021020 con 0x7f0d3812ef00 2023-09-30T01:55:56.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:56.028+0000 7f0d367fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:56.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:56.028+0000 7f0d367fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:55:56.029 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:55:56.029 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:55:56.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:56.030+0000 7f0d40293700 1 -- 172.21.15.7:0/2713514324 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f0d240357e0 msgr2=0x7f0d24037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:56.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:56.030+0000 7f0d40293700 1 --2- 172.21.15.7:0/2713514324 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f0d240357e0 0x7f0d24037c90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0d28006fd0 tx=0x7f0d28008040 comp rx=0 tx=0).stop 2023-09-30T01:55:56.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:56.030+0000 7f0d40293700 10 monclient: shutdown 2023-09-30T01:55:56.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:56.030+0000 7f0d40293700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:56.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:56.030+0000 7f0d40293700 1 -- 172.21.15.7:0/2713514324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d3812ef00 msgr2=0x7f0d3812dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:56.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:56.030+0000 7f0d40293700 1 --2- 172.21.15.7:0/2713514324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d3812ef00 0x7f0d3812dfc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0d381331d0 tx=0x7f0d30018000 comp rx=0 tx=0).stop 2023-09-30T01:55:56.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:56.030+0000 7f0d40293700 1 -- 172.21.15.7:0/2713514324 shutdown_connections 2023-09-30T01:55:56.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:56.030+0000 7f0d40293700 1 --2- 172.21.15.7:0/2713514324 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f0d240357e0 0x7f0d24037c90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:56.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:56.030+0000 7f0d40293700 1 --2- 172.21.15.7:0/2713514324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0d3812ef00 0x7f0d3812dfc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:56.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:56.030+0000 7f0d40293700 1 --2- 172.21.15.7:0/2713514324 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0d38096a50 0x7f0d380a0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:56.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:56.030+0000 7f0d40293700 1 --2- 172.21.15.7:0/2713514324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d38096e20 0x7f0d381339d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:56.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:56.030+0000 7f0d40293700 1 -- 172.21.15.7:0/2713514324 >> 172.21.15.7:0/2713514324 conn(0x7f0d38070cf0 msgr2=0x7f0d380a4800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:56.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:56.030+0000 7f0d40293700 1 -- 172.21.15.7:0/2713514324 shutdown_connections 2023-09-30T01:55:56.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:56.030+0000 7f0d40293700 1 -- 172.21.15.7:0/2713514324 wait complete. 2023-09-30T01:55:59.043 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:55:59.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.173+0000 7fb1fdbaa700 10 monclient: get_monmap_and_config 2023-09-30T01:55:59.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.173+0000 7fb1fdbaa700 10 monclient: build_initial_monmap 2023-09-30T01:55:59.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.173+0000 7fb1fdbaa700 10 monclient: monmap: 2023-09-30T01:55:59.174 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:59.174 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:59.174 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:55:59.174777+0000 2023-09-30T01:55:59.174 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:55:59.174777+0000 2023-09-30T01:55:59.174 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:59.174 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:59.174 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:55:59.174 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:55:59.174 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:55:59.175 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:55:59.175 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:55:59.175 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:59.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.174+0000 7fb1fdbaa700 1 Processor -- start 2023-09-30T01:55:59.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.174+0000 7fb1fdbaa700 1 -- start start 2023-09-30T01:55:59.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.174+0000 7fb1fdbaa700 10 monclient: init 2023-09-30T01:55:59.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.174+0000 7fb1fdbaa700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:59.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.174+0000 7fb1fdbaa700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-09-30T01:55:59.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.174+0000 7fb1fdbaa700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb1f806f550 0x7fb1f809af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:59.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.174+0000 7fb1fdbaa700 10 monclient(hunting): picked mon.noname-b con 0x7fb1f806f550 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:59.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.174+0000 7fb1fdbaa700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb1f8069b70 0x7fb1f809dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:59.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.174+0000 7fb1fdbaa700 10 monclient(hunting): picked mon.noname-c con 0x7fb1f8069b70 addr v2:172.21.15.50:3300/0 2023-09-30T01:55:59.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.174+0000 7fb1fdbaa700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb1f806f920 addr v1:172.21.15.7:6789/0 2023-09-30T01:55:59.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.174+0000 7fb1fdbaa700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f806aee0 con 0x7fb1f806f920 2023-09-30T01:55:59.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.174+0000 7fb1fdbaa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:59.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.174+0000 7fb1fdbaa700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f80699a0 con 0x7fb1f8069b70 2023-09-30T01:55:59.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.174+0000 7fb1fdbaa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:59.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.174+0000 7fb1fdbaa700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f806b1c0 con 0x7fb1f806f550 2023-09-30T01:55:59.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.174+0000 7fb1fdbaa700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:59.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1fdbaa700 10 monclient(hunting): authenticate will time out at 1248.240008s 2023-09-30T01:55:59.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1fcba8700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb1f806f550 0x7fb1f809af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:59.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1fd3a9700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb1f806f920 0x7fb1f809e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45612/0 (socket says 172.21.15.7:45612) 2023-09-30T01:55:59.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1fd3a9700 1 -- 172.21.15.7:0/54465139 learned_addr learned my addr 172.21.15.7:0/54465139 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:55:59.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1fcba8700 10 monclient(hunting): get_auth_request con 0x7fb1f806f550 auth_method 0 2023-09-30T01:55:59.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1fcba8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:59.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1fcba8700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:59.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1fcba8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:59.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1fcba8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:59.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1fcba8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:59.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1fcba8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:59.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1f7fff700 1 --2- 172.21.15.7:0/54465139 >> v2:172.21.15.50:3300/0 conn(0x7fb1f8069b70 0x7fb1f809dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:59.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1fcba8700 10 monclient(hunting): handle_auth_done global_id 4409 payload 274 2023-09-30T01:55:59.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1fcba8700 1 -- 172.21.15.7:0/54465139 >> v2:172.21.15.50:3300/0 conn(0x7fb1f8069b70 msgr2=0x7fb1f809dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:59.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1fcba8700 1 --2- 172.21.15.7:0/54465139 >> v2:172.21.15.50:3300/0 conn(0x7fb1f8069b70 0x7fb1f809dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:59.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1fcba8700 1 -- 172.21.15.7:0/54465139 >> v1:172.21.15.7:6789/0 conn(0x7fb1f806f920 legacy=0x7fb1f809e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:55:59.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1fcba8700 10 monclient: _finish_hunting 0 2023-09-30T01:55:59.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1fcba8700 1 monclient: found mon.noname-b 2023-09-30T01:55:59.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1fcba8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:59.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1fcba8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:55:59.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1fcba8700 1 -- 172.21.15.7:0/54465139 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f809b4b0 con 0x7fb1f806f550 2023-09-30T01:55:59.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1f7fff700 10 monclient: get_auth_request con 0x7fb1f8069b70 auth_method 0 2023-09-30T01:55:59.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.175+0000 7fb1f7fff700 1 --2- 172.21.15.7:0/54465139 >> v2:172.21.15.50:3300/0 conn(0x7fb1f8069b70 0x7fb1f809dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:55:59.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.177+0000 7fb1fcba8700 1 -- 172.21.15.7:0/54465139 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb1f806f550 msgr2=0x7fb1f809af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:59.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.177+0000 7fb1fcba8700 1 --2- 172.21.15.7:0/54465139 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb1f806f550 0x7fb1f809af70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec0098e0 tx=0x7fb1ec0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce09e9692151972f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:59.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.177+0000 7fb1f77fe700 1 -- 172.21.15.7:0/54465139 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec00ba30 con 0x7fb1f806f550 2023-09-30T01:55:59.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.177+0000 7fb1f77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:59.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.177+0000 7fb1f77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:55:59.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.177+0000 7fb1f77fe700 10 monclient: dump: 2023-09-30T01:55:59.183 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:59.183 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:59.183 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:59.183 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:59.183 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:59.183 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:59.183 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:59.183 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:59.183 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:59.184 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:59.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.177+0000 7fb1f77fe700 10 monclient: _finish_auth 0 2023-09-30T01:55:59.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.177+0000 7fb1f77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:59.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.177+0000 7fb1fdbaa700 5 monclient: authenticate success, global_id 4409 2023-09-30T01:55:59.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.178+0000 7fb1fdbaa700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:55:59.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.177+0000 7fb1f77fe700 1 -- 172.21.15.7:0/54465139 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1ec00bb90 con 0x7fb1f806f550 2023-09-30T01:55:59.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.178+0000 7fb1f77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:59.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.178+0000 7fb1fdbaa700 10 monclient: get_monmap_and_config success 2023-09-30T01:55:59.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.178+0000 7fb1fdbaa700 4 set_mon_vals no callback set 2023-09-30T01:55:59.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.178+0000 7fb1f77fe700 1 -- 172.21.15.7:0/54465139 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec011420 con 0x7fb1f806f550 2023-09-30T01:55:59.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.178+0000 7fb1f77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:59.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.178+0000 7fb1f77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:55:59.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.178+0000 7fb1f77fe700 10 monclient: dump: 2023-09-30T01:55:59.185 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:59.185 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:59.185 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:59.185 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:59.185 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:59.185 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:59.185 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:59.185 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:59.185 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:59.185 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:59.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.178+0000 7fb1fdbaa700 10 monclient: shutdown 2023-09-30T01:55:59.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.178+0000 7fb1fdbaa700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:59.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.178+0000 7fb1fdbaa700 1 -- 172.21.15.7:0/54465139 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb1f806f550 msgr2=0x7fb1f809af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:59.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.178+0000 7fb1fdbaa700 1 --2- 172.21.15.7:0/54465139 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb1f806f550 0x7fb1f809af70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec0098e0 tx=0x7fb1ec0091a0 comp rx=0 tx=0).stop 2023-09-30T01:55:59.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.178+0000 7fb1fdbaa700 1 -- 172.21.15.7:0/54465139 shutdown_connections 2023-09-30T01:55:59.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.178+0000 7fb1fdbaa700 1 --2- 172.21.15.7:0/54465139 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb1f806f550 0x7fb1f809af70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:59.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.178+0000 7fb1fdbaa700 1 --2- 172.21.15.7:0/54465139 >> v2:172.21.15.50:3300/0 conn(0x7fb1f8069b70 0x7fb1f809dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:59.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.178+0000 7fb1fdbaa700 1 -- 172.21.15.7:0/54465139 >> 172.21.15.7:0/54465139 conn(0x7fb1f80a32a0 msgr2=0x7fb1f80a36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:59.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.178+0000 7fb1fdbaa700 1 -- 172.21.15.7:0/54465139 shutdown_connections 2023-09-30T01:55:59.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.178+0000 7fb1fdbaa700 1 -- 172.21.15.7:0/54465139 wait complete. 2023-09-30T01:55:59.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.179+0000 7fb1fdbaa700 10 monclient: build_initial_monmap 2023-09-30T01:55:59.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.179+0000 7fb1fdbaa700 10 monclient: monmap: 2023-09-30T01:55:59.187 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:55:59.187 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:55:59.187 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:55:59.187 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:55:59.187 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:55:59.187 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:59.187 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:55:59.187 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:55:59.187 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:55:59.187 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:59.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.179+0000 7fb1fdbaa700 1 Processor -- start 2023-09-30T01:55:59.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.179+0000 7fb1fdbaa700 1 -- start start 2023-09-30T01:55:59.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.179+0000 7fb1fdbaa700 10 monclient: init 2023-09-30T01:55:59.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.179+0000 7fb1fdbaa700 10 monclient: _reopen_session rank -1 2023-09-30T01:55:59.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.179+0000 7fb1fdbaa700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:55:59.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fdbaa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1f8069b70 0x7fb1f813e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:59.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fdbaa700 10 monclient(hunting): picked mon.noname-a con 0x7fb1f8069b70 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:55:59.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fdbaa700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1f806f550 0x7fb1f813d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:59.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fdbaa700 10 monclient(hunting): picked mon.noname-b con 0x7fb1f806f550 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:55:59.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fdbaa700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb1f806f920 0x7fb1f8141820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:59.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fdbaa700 10 monclient(hunting): picked mon.noname-c con 0x7fb1f806f920 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:55:59.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fdbaa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:59.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fdbaa700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f8122ce0 con 0x7fb1f8069b70 2023-09-30T01:55:59.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fdbaa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:59.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fdbaa700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f8120fd0 con 0x7fb1f806f550 2023-09-30T01:55:59.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fdbaa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:55:59.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fdbaa700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f8122a00 con 0x7fb1f806f920 2023-09-30T01:55:59.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fdbaa700 10 monclient(hunting): _renew_subs 2023-09-30T01:55:59.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fdbaa700 10 monclient(hunting): authenticate will time out at 1248.245072s 2023-09-30T01:55:59.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fcba8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1f8069b70 0x7fb1f813e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:59.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fd3a9700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb1f806f920 0x7fb1f8141820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:59.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fcba8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1f8069b70 0x7fb1f813e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38958/0 (socket says 172.21.15.7:38958) 2023-09-30T01:55:59.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fd3a9700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb1f806f920 0x7fb1f8141820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:49888/0 (socket says 172.21.15.7:49888) 2023-09-30T01:55:59.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fcba8700 1 -- 172.21.15.7:0/267287981 learned_addr learned my addr 172.21.15.7:0/267287981 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:55:59.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fcba8700 10 monclient(hunting): get_auth_request con 0x7fb1f8069b70 auth_method 0 2023-09-30T01:55:59.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fcba8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:59.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fcba8700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:59.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fcba8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:59.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fd3a9700 10 monclient(hunting): get_auth_request con 0x7fb1f806f920 auth_method 0 2023-09-30T01:55:59.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fd3a9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:59.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fd3a9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:59.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fd3a9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:59.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1f7fff700 1 --2- 172.21.15.7:0/267287981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1f806f550 0x7fb1f813d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:59.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fd3a9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:59.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fd3a9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:59.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fd3a9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:59.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fcba8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:59.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fcba8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:59.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fcba8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:59.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1f7fff700 10 monclient(hunting): get_auth_request con 0x7fb1f806f550 auth_method 0 2023-09-30T01:55:59.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1f7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:55:59.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1f7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:55:59.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1f7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:55:59.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1f7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:55:59.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1f7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:55:59.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1f7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:55:59.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fd3a9700 10 monclient(hunting): handle_auth_done global_id 4412 payload 824 2023-09-30T01:55:59.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fd3a9700 1 -- 172.21.15.7:0/267287981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1f806f550 msgr2=0x7fb1f813d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:59.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fd3a9700 1 --2- 172.21.15.7:0/267287981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1f806f550 0x7fb1f813d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:59.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fd3a9700 1 -- 172.21.15.7:0/267287981 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1f8069b70 msgr2=0x7fb1f813e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:59.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fd3a9700 1 --2- 172.21.15.7:0/267287981 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1f8069b70 0x7fb1f813e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:59.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fd3a9700 10 monclient: _finish_hunting 0 2023-09-30T01:55:59.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fd3a9700 1 monclient: found mon.noname-c 2023-09-30T01:55:59.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fd3a9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:55:59.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fd3a9700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:55:59.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fd3a9700 1 -- 172.21.15.7:0/267287981 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ec009420 con 0x7fb1f806f920 2023-09-30T01:55:59.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.180+0000 7fb1fcba8700 1 --2- 172.21.15.7:0/267287981 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1f8069b70 0x7fb1f813e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:55:59.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.182+0000 7fb1fd3a9700 1 -- 172.21.15.7:0/267287981 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb1f806f920 msgr2=0x7fb1f8141820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:55:59.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.182+0000 7fb1fd3a9700 1 --2- 172.21.15.7:0/267287981 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb1f806f920 0x7fb1f8141820 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb1f000cb60 tx=0x7fb1f0005340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:59.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.182+0000 7fb1f4ff9700 1 -- 172.21.15.7:0/267287981 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f000a9f0 con 0x7fb1f806f920 2023-09-30T01:55:59.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.182+0000 7fb1f4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:59.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.182+0000 7fb1f4ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:55:59.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.182+0000 7fb1f4ff9700 10 monclient: dump: 2023-09-30T01:55:59.197 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:59.197 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:59.197 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:59.198 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:59.198 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:59.198 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:59.198 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:59.198 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:59.198 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:59.198 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:59.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.182+0000 7fb1f4ff9700 10 monclient: _finish_auth 0 2023-09-30T01:55:59.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.182+0000 7fb1f4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:55:59.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.182+0000 7fb1f4ff9700 1 -- 172.21.15.7:0/267287981 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f000ab70 con 0x7fb1f806f920 2023-09-30T01:55:59.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.182+0000 7fb1fdbaa700 5 monclient: authenticate success, global_id 4412 2023-09-30T01:55:59.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.182+0000 7fb1fdbaa700 10 monclient: _renew_subs 2023-09-30T01:55:59.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.182+0000 7fb1fdbaa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:59.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.182+0000 7fb1fdbaa700 1 -- 172.21.15.7:0/267287981 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f8122fc0 con 0x7fb1f806f920 2023-09-30T01:55:59.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.182+0000 7fb1f4ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:55:59.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.182+0000 7fb1f4ff9700 1 -- 172.21.15.7:0/267287981 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f00070f0 con 0x7fb1f806f920 2023-09-30T01:55:59.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.182+0000 7fb1f4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:55:59.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.182+0000 7fb1f4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:55:59.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.182+0000 7fb1f4ff9700 10 monclient: dump: 2023-09-30T01:55:59.199 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:55:59.199 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:55:59.199 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:59.199 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:55:59.199 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:55:59.199 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:55:59.200 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:55:59.200 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:55:59.200 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:55:59.200 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:55:59.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.182+0000 7fb1fdbaa700 10 monclient: _renew_subs 2023-09-30T01:55:59.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.182+0000 7fb1f6ffd700 4 set_mon_vals no callback set 2023-09-30T01:55:59.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.183+0000 7fb1fdbaa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:59.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.183+0000 7fb1fdbaa700 1 -- 172.21.15.7:0/267287981 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f813dea0 con 0x7fb1f806f920 2023-09-30T01:55:59.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.183+0000 7fb1fdbaa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:59.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.183+0000 7fb1fdbaa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:59.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.183+0000 7fb1fdbaa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:59.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.183+0000 7fb1f4ff9700 1 -- 172.21.15.7:0/267287981 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fb1f00074e0 con 0x7fb1f806f920 2023-09-30T01:55:59.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.183+0000 7fb1fdbaa700 1 -- 172.21.15.7:0/267287981 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f813e120 con 0x7fb1f806f920 2023-09-30T01:55:59.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.184+0000 7fb1f4ff9700 1 --2- 172.21.15.7:0/267287981 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb1dc0357e0 0x7fb1dc037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:55:59.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.184+0000 7fb1f4ff9700 1 -- 172.21.15.7:0/267287981 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fb1f0047fd0 con 0x7fb1f806f920 2023-09-30T01:55:59.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.184+0000 7fb1fcba8700 1 --2- 172.21.15.7:0/267287981 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb1dc0357e0 0x7fb1dc037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:55:59.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.184+0000 7fb1fcba8700 10 monclient: get_auth_request con 0x7fb1dc0357e0 auth_method 0 2023-09-30T01:55:59.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.186+0000 7fb1fcba8700 1 --2- 172.21.15.7:0/267287981 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb1dc0357e0 0x7fb1dc037c90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec0030c0 tx=0x7fb1ec016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:55:59.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.188+0000 7fb1f4ff9700 1 -- 172.21.15.7:0/267287981 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb1f0010950 con 0x7fb1f806f920 2023-09-30T01:55:59.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.188+0000 7fb1f4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:55:59.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.188+0000 7fb1f4ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:55:59.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.425+0000 7fb1fdbaa700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:55:59.425 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.425+0000 7fb1fdbaa700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:59.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.425+0000 7fb1fdbaa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:55:59.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.425+0000 7fb1fdbaa700 1 -- 172.21.15.7:0/267287981 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb1f8064c10 con 0x7fb1f806f920 2023-09-30T01:55:59.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.425+0000 7fb1f4ff9700 1 -- 172.21.15.7:0/267287981 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb1f0010300 con 0x7fb1f806f920 2023-09-30T01:55:59.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.425+0000 7fb1f4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:55:59.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.425+0000 7fb1f4ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:55:59.427 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:55:59.427 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:55:59.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.427+0000 7fb1fdbaa700 1 -- 172.21.15.7:0/267287981 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb1dc0357e0 msgr2=0x7fb1dc037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:59.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.427+0000 7fb1fdbaa700 1 --2- 172.21.15.7:0/267287981 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb1dc0357e0 0x7fb1dc037c90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec0030c0 tx=0x7fb1ec016000 comp rx=0 tx=0).stop 2023-09-30T01:55:59.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.428+0000 7fb1fdbaa700 10 monclient: shutdown 2023-09-30T01:55:59.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.428+0000 7fb1fdbaa700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:55:59.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.428+0000 7fb1fdbaa700 1 -- 172.21.15.7:0/267287981 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb1f806f920 msgr2=0x7fb1f8141820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:55:59.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.428+0000 7fb1fdbaa700 1 --2- 172.21.15.7:0/267287981 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb1f806f920 0x7fb1f8141820 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb1f000cb60 tx=0x7fb1f0005340 comp rx=0 tx=0).stop 2023-09-30T01:55:59.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.428+0000 7fb1fdbaa700 1 -- 172.21.15.7:0/267287981 shutdown_connections 2023-09-30T01:55:59.428 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.428+0000 7fb1fdbaa700 1 --2- 172.21.15.7:0/267287981 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb1dc0357e0 0x7fb1dc037c90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:59.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.428+0000 7fb1fdbaa700 1 --2- 172.21.15.7:0/267287981 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1f8069b70 0x7fb1f813e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:59.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.428+0000 7fb1fdbaa700 1 --2- 172.21.15.7:0/267287981 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb1f806f920 0x7fb1f8141820 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:59.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.428+0000 7fb1fdbaa700 1 --2- 172.21.15.7:0/267287981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1f806f550 0x7fb1f813d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:55:59.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.428+0000 7fb1fdbaa700 1 -- 172.21.15.7:0/267287981 >> 172.21.15.7:0/267287981 conn(0x7fb1f80a32a0 msgr2=0x7fb1f80993d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:55:59.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.428+0000 7fb1fdbaa700 1 -- 172.21.15.7:0/267287981 shutdown_connections 2023-09-30T01:55:59.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:55:59.428+0000 7fb1fdbaa700 1 -- 172.21.15.7:0/267287981 wait complete. 2023-09-30T01:56:02.440 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:56:02.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.571+0000 7fdd6b2f0700 10 monclient: get_monmap_and_config 2023-09-30T01:56:02.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.571+0000 7fdd6b2f0700 10 monclient: build_initial_monmap 2023-09-30T01:56:02.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.571+0000 7fdd6b2f0700 10 monclient: monmap: 2023-09-30T01:56:02.571 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:02.571 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:56:02.572378+0000 2023-09-30T01:56:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:56:02.572378+0000 2023-09-30T01:56:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:56:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:56:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:56:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:56:02.572 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:56:02.572 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:02.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.571+0000 7fdd6b2f0700 1 Processor -- start 2023-09-30T01:56:02.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6b2f0700 1 -- start start 2023-09-30T01:56:02.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6b2f0700 10 monclient: init 2023-09-30T01:56:02.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6b2f0700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:02.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6b2f0700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-09-30T01:56:02.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6b2f0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdd64096a50 addr v1:172.21.15.7:6789/0 2023-09-30T01:56:02.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6b2f0700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdd640a2390 0x7fdd64096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:02.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6b2f0700 10 monclient(hunting): picked mon.noname-b con 0x7fdd640a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:02.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6b2f0700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdd64097360 addr v1:172.21.15.50:6789/0 2023-09-30T01:56:02.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6b2f0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd640980e0 con 0x7fdd64096a50 2023-09-30T01:56:02.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6b2f0700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd64132eb0 con 0x7fdd64097360 2023-09-30T01:56:02.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6b2f0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:02.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6b2f0700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd640991a0 con 0x7fdd640a2390 2023-09-30T01:56:02.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6b2f0700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:02.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6b2f0700 10 monclient(hunting): authenticate will time out at 1251.637653s 2023-09-30T01:56:02.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6908c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdd64096a50 0x7fdd641327a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45624/0 (socket says 172.21.15.7:45624) 2023-09-30T01:56:02.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6888b700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdd640a2390 0x7fdd64096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:02.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6908c700 1 -- 172.21.15.7:0/2196931985 learned_addr learned my addr 172.21.15.7:0/2196931985 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:56:02.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6888b700 10 monclient(hunting): get_auth_request con 0x7fdd640a2390 auth_method 0 2023-09-30T01:56:02.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6888b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:02.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6888b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:02.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6888b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6888b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6888b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.572+0000 7fdd6888b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.573+0000 7fdd6888b700 10 monclient(hunting): handle_auth_done global_id 4415 payload 274 2023-09-30T01:56:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.573+0000 7fdd6888b700 1 -- 172.21.15.7:0/2196931985 >> v1:172.21.15.50:6789/0 conn(0x7fdd64097360 legacy=0x7fdd64139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.573+0000 7fdd6888b700 1 -- 172.21.15.7:0/2196931985 >> v1:172.21.15.7:6789/0 conn(0x7fdd64096a50 legacy=0x7fdd641327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.573+0000 7fdd6888b700 10 monclient: _finish_hunting 0 2023-09-30T01:56:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.573+0000 7fdd6888b700 1 monclient: found mon.noname-b 2023-09-30T01:56:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.573+0000 7fdd6888b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:02.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.573+0000 7fdd6888b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:56:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.573+0000 7fdd6888b700 1 -- 172.21.15.7:0/2196931985 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd64099480 con 0x7fdd640a2390 2023-09-30T01:56:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd6888b700 1 -- 172.21.15.7:0/2196931985 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdd640a2390 msgr2=0x7fdd64096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd6888b700 1 --2- 172.21.15.7:0/2196931985 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdd640a2390 0x7fdd64096e20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdd58009910 tx=0x7fdd580091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d97cdbc7c2ab70f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd63fff700 1 -- 172.21.15.7:0/2196931985 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5800b9f0 con 0x7fdd640a2390 2023-09-30T01:56:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd63fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd63fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:56:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd63fff700 10 monclient: dump: 2023-09-30T01:56:02.581 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:02.582 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd63fff700 10 monclient: _finish_auth 0 2023-09-30T01:56:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd63fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd63fff700 1 -- 172.21.15.7:0/2196931985 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5800bb50 con 0x7fdd640a2390 2023-09-30T01:56:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd6b2f0700 5 monclient: authenticate success, global_id 4415 2023-09-30T01:56:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd6b2f0700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:56:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd63fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd6b2f0700 10 monclient: get_monmap_and_config success 2023-09-30T01:56:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd63fff700 1 -- 172.21.15.7:0/2196931985 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd58011420 con 0x7fdd640a2390 2023-09-30T01:56:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd6b2f0700 4 set_mon_vals no callback set 2023-09-30T01:56:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd63fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd63fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:56:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd63fff700 10 monclient: dump: 2023-09-30T01:56:02.584 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:02.584 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:02.584 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:02.584 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:02.584 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:02.584 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:02.584 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:02.584 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:02.584 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:02.584 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:02.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd6b2f0700 10 monclient: shutdown 2023-09-30T01:56:02.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd6b2f0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:02.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd6b2f0700 1 -- 172.21.15.7:0/2196931985 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdd640a2390 msgr2=0x7fdd64096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:02.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.575+0000 7fdd6b2f0700 1 --2- 172.21.15.7:0/2196931985 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdd640a2390 0x7fdd64096e20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdd58009910 tx=0x7fdd580091a0 comp rx=0 tx=0).stop 2023-09-30T01:56:02.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.576+0000 7fdd6b2f0700 1 -- 172.21.15.7:0/2196931985 shutdown_connections 2023-09-30T01:56:02.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.576+0000 7fdd6b2f0700 1 --2- 172.21.15.7:0/2196931985 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdd640a2390 0x7fdd64096e20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:02.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.576+0000 7fdd6b2f0700 1 -- 172.21.15.7:0/2196931985 >> 172.21.15.7:0/2196931985 conn(0x7fdd64070cf0 msgr2=0x7fdd640710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:02.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.576+0000 7fdd6b2f0700 1 -- 172.21.15.7:0/2196931985 shutdown_connections 2023-09-30T01:56:02.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.576+0000 7fdd6b2f0700 1 -- 172.21.15.7:0/2196931985 wait complete. 2023-09-30T01:56:02.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.576+0000 7fdd6b2f0700 10 monclient: build_initial_monmap 2023-09-30T01:56:02.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.576+0000 7fdd6b2f0700 10 monclient: monmap: 2023-09-30T01:56:02.585 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:56:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:56:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:56:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:56:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:56:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:56:02.586 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.576+0000 7fdd6b2f0700 1 Processor -- start 2023-09-30T01:56:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6b2f0700 1 -- start start 2023-09-30T01:56:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6b2f0700 10 monclient: init 2023-09-30T01:56:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6b2f0700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6b2f0700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:56:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6b2f0700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdd64096a50 0x7fdd6409b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6b2f0700 10 monclient(hunting): picked mon.noname-c con 0x7fdd64096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6b2f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd64097360 0x7fdd6409be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6b2f0700 10 monclient(hunting): picked mon.noname-b con 0x7fdd64097360 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:56:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6b2f0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd640a2390 0x7fdd6409c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6b2f0700 10 monclient(hunting): picked mon.noname-a con 0x7fdd640a2390 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:56:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6b2f0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6b2f0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd640a1330 con 0x7fdd640a2390 2023-09-30T01:56:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6b2f0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6b2f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd64132270 con 0x7fdd64097360 2023-09-30T01:56:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6b2f0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6b2f0700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6409f0b0 con 0x7fdd64096a50 2023-09-30T01:56:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6b2f0700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6b2f0700 10 monclient(hunting): authenticate will time out at 1251.642632s 2023-09-30T01:56:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6908c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdd64096a50 0x7fdd6409b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6908c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdd64096a50 0x7fdd6409b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:49906/0 (socket says 172.21.15.7:49906) 2023-09-30T01:56:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6908c700 1 -- 172.21.15.7:0/1787955113 learned_addr learned my addr 172.21.15.7:0/1787955113 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6908c700 10 monclient(hunting): get_auth_request con 0x7fdd64096a50 auth_method 0 2023-09-30T01:56:02.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6908c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:02.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6908c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:02.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6908c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:02.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6988d700 1 --2- 172.21.15.7:0/1787955113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd640a2390 0x7fdd6409c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:02.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6988d700 10 monclient(hunting): get_auth_request con 0x7fdd640a2390 auth_method 0 2023-09-30T01:56:02.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6988d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:02.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6988d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:02.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6988d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:02.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6908c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:02.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6908c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:02.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6908c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:02.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6988d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:02.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6988d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:02.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6988d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:02.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6888b700 1 --2- 172.21.15.7:0/1787955113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd64097360 0x7fdd6409be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:02.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6888b700 10 monclient(hunting): get_auth_request con 0x7fdd64097360 auth_method 0 2023-09-30T01:56:02.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6888b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:02.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6888b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:02.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.577+0000 7fdd6888b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:02.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.578+0000 7fdd6908c700 10 monclient(hunting): handle_auth_done global_id 4418 payload 824 2023-09-30T01:56:02.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.578+0000 7fdd6908c700 1 -- 172.21.15.7:0/1787955113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd64097360 msgr2=0x7fdd6409be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:02.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.578+0000 7fdd6908c700 1 --2- 172.21.15.7:0/1787955113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd64097360 0x7fdd6409be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:02.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.578+0000 7fdd6908c700 1 -- 172.21.15.7:0/1787955113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd640a2390 msgr2=0x7fdd6409c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:02.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.578+0000 7fdd6908c700 1 --2- 172.21.15.7:0/1787955113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd640a2390 0x7fdd6409c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:02.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.578+0000 7fdd6908c700 10 monclient: _finish_hunting 0 2023-09-30T01:56:02.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.578+0000 7fdd6908c700 1 monclient: found mon.noname-c 2023-09-30T01:56:02.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.578+0000 7fdd6908c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:02.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.578+0000 7fdd6908c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:56:02.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.578+0000 7fdd6908c700 1 -- 172.21.15.7:0/1787955113 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd58009420 con 0x7fdd64096a50 2023-09-30T01:56:02.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.578+0000 7fdd6988d700 1 --2- 172.21.15.7:0/1787955113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd640a2390 0x7fdd6409c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:56:02.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd6908c700 1 -- 172.21.15.7:0/1787955113 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdd64096a50 msgr2=0x7fdd6409b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:02.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd6908c700 1 --2- 172.21.15.7:0/1787955113 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdd64096a50 0x7fdd6409b900 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdd5400ccd0 tx=0x7fdd54005480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:02.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd617fa700 1 -- 172.21.15.7:0/1787955113 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5400ab90 con 0x7fdd64096a50 2023-09-30T01:56:02.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd617fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:02.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd617fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:56:02.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd617fa700 10 monclient: dump: 2023-09-30T01:56:02.596 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:02.596 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:02.597 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:02.597 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:02.597 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:02.597 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:02.597 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:02.597 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:02.597 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:02.597 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:02.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd617fa700 10 monclient: _finish_auth 0 2023-09-30T01:56:02.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd617fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:02.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd617fa700 1 -- 172.21.15.7:0/1787955113 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5400acf0 con 0x7fdd64096a50 2023-09-30T01:56:02.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd6b2f0700 5 monclient: authenticate success, global_id 4418 2023-09-30T01:56:02.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd6b2f0700 10 monclient: _renew_subs 2023-09-30T01:56:02.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd6b2f0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:02.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd6b2f0700 1 -- 172.21.15.7:0/1787955113 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd6409d3e0 con 0x7fdd64096a50 2023-09-30T01:56:02.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd617fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:02.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd617fa700 1 -- 172.21.15.7:0/1787955113 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd54007340 con 0x7fdd64096a50 2023-09-30T01:56:02.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd617fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:02.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd617fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:56:02.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd617fa700 10 monclient: dump: 2023-09-30T01:56:02.599 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:02.599 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:02.599 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:02.599 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:02.599 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:02.599 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:02.599 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:02.599 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:02.599 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:02.600 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:02.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd6b2f0700 10 monclient: _renew_subs 2023-09-30T01:56:02.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd6b2f0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:02.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd6b2f0700 1 -- 172.21.15.7:0/1787955113 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd6409dee0 con 0x7fdd64096a50 2023-09-30T01:56:02.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.580+0000 7fdd637fe700 4 set_mon_vals no callback set 2023-09-30T01:56:02.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.581+0000 7fdd617fa700 1 -- 172.21.15.7:0/1787955113 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fdd54007730 con 0x7fdd64096a50 2023-09-30T01:56:02.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.581+0000 7fdd6b2f0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:02.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.581+0000 7fdd6b2f0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:02.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.581+0000 7fdd6b2f0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:02.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.581+0000 7fdd6b2f0700 1 -- 172.21.15.7:0/1787955113 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd64137a30 con 0x7fdd64096a50 2023-09-30T01:56:02.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.581+0000 7fdd617fa700 1 --2- 172.21.15.7:0/1787955113 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fdd500357e0 0x7fdd50037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:02.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.581+0000 7fdd617fa700 1 -- 172.21.15.7:0/1787955113 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fdd540481f0 con 0x7fdd64096a50 2023-09-30T01:56:02.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.581+0000 7fdd6888b700 1 --2- 172.21.15.7:0/1787955113 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fdd500357e0 0x7fdd50037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:02.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.581+0000 7fdd6888b700 10 monclient: get_auth_request con 0x7fdd500357e0 auth_method 0 2023-09-30T01:56:02.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.584+0000 7fdd6888b700 1 --2- 172.21.15.7:0/1787955113 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fdd500357e0 0x7fdd50037c90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdd6406a090 tx=0x7fdd58016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:02.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.586+0000 7fdd617fa700 1 -- 172.21.15.7:0/1787955113 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdd5400ae60 con 0x7fdd64096a50 2023-09-30T01:56:02.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.586+0000 7fdd617fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:02.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.586+0000 7fdd617fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:56:02.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.822+0000 7fdd6b2f0700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:56:02.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.822+0000 7fdd6b2f0700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:02.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.822+0000 7fdd6b2f0700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:02.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.822+0000 7fdd6b2f0700 1 -- 172.21.15.7:0/1787955113 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdd64064c10 con 0x7fdd64096a50 2023-09-30T01:56:02.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.822+0000 7fdd617fa700 1 -- 172.21.15.7:0/1787955113 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fdd54010300 con 0x7fdd64096a50 2023-09-30T01:56:02.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.822+0000 7fdd617fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:02.822 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.822+0000 7fdd617fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:56:02.823 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:56:02.823 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:56:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.824+0000 7fdd6b2f0700 1 -- 172.21.15.7:0/1787955113 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fdd500357e0 msgr2=0x7fdd50037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.824+0000 7fdd6b2f0700 1 --2- 172.21.15.7:0/1787955113 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fdd500357e0 0x7fdd50037c90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdd6406a090 tx=0x7fdd58016000 comp rx=0 tx=0).stop 2023-09-30T01:56:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.824+0000 7fdd6b2f0700 10 monclient: shutdown 2023-09-30T01:56:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.824+0000 7fdd6b2f0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.824+0000 7fdd6b2f0700 1 -- 172.21.15.7:0/1787955113 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdd64096a50 msgr2=0x7fdd6409b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.824+0000 7fdd6b2f0700 1 --2- 172.21.15.7:0/1787955113 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdd64096a50 0x7fdd6409b900 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdd5400ccd0 tx=0x7fdd54005480 comp rx=0 tx=0).stop 2023-09-30T01:56:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.824+0000 7fdd6b2f0700 1 -- 172.21.15.7:0/1787955113 shutdown_connections 2023-09-30T01:56:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.824+0000 7fdd6b2f0700 1 --2- 172.21.15.7:0/1787955113 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fdd500357e0 0x7fdd50037c90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.824+0000 7fdd6b2f0700 1 --2- 172.21.15.7:0/1787955113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd640a2390 0x7fdd6409c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.824+0000 7fdd6b2f0700 1 --2- 172.21.15.7:0/1787955113 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fdd64096a50 0x7fdd6409b900 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.824+0000 7fdd6b2f0700 1 --2- 172.21.15.7:0/1787955113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdd64097360 0x7fdd6409be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.824+0000 7fdd6b2f0700 1 -- 172.21.15.7:0/1787955113 >> 172.21.15.7:0/1787955113 conn(0x7fdd64070cf0 msgr2=0x7fdd641336f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.824+0000 7fdd6b2f0700 1 -- 172.21.15.7:0/1787955113 shutdown_connections 2023-09-30T01:56:02.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:02.824+0000 7fdd6b2f0700 1 -- 172.21.15.7:0/1787955113 wait complete. 2023-09-30T01:56:05.837 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:56:05.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.968+0000 7f98ec73c700 10 monclient: get_monmap_and_config 2023-09-30T01:56:05.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.968+0000 7f98ec73c700 10 monclient: build_initial_monmap 2023-09-30T01:56:05.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.968+0000 7f98ec73c700 10 monclient: monmap: 2023-09-30T01:56:05.968 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:05.968 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:05.968 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:56:05.969316+0000 2023-09-30T01:56:05.969 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:56:05.969316+0000 2023-09-30T01:56:05.969 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:05.969 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:05.969 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:56:05.969 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:56:05.969 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:56:05.969 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:56:05.969 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:56:05.969 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:05.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.968+0000 7f98ec73c700 1 Processor -- start 2023-09-30T01:56:05.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.968+0000 7f98ec73c700 1 -- start start 2023-09-30T01:56:05.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.968+0000 7f98ec73c700 10 monclient: init 2023-09-30T01:56:05.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98ec73c700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:05.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98ec73c700 10 monclient: _add_conns ranks=[0,2,4,3,1] 2023-09-30T01:56:05.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98ec73c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f98e4096a50 addr v1:172.21.15.7:6789/0 2023-09-30T01:56:05.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98ec73c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f98e4096e20 0x7f98e40971f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:05.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98ec73c700 10 monclient(hunting): picked mon.noname-b con 0x7f98e4096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:05.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98ec73c700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f98e4137710 0x7f98e4139af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:05.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98ec73c700 10 monclient(hunting): picked mon.noname-c con 0x7f98e4137710 addr v2:172.21.15.50:3300/0 2023-09-30T01:56:05.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98ec73c700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e4132d90 con 0x7f98e4096a50 2023-09-30T01:56:05.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98ec73c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:05.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98ec73c700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98e4097b40 con 0x7f98e4137710 2023-09-30T01:56:05.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98ec73c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:05.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98ec73c700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e4099160 con 0x7f98e4096e20 2023-09-30T01:56:05.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98ec73c700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:05.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98ec73c700 10 monclient(hunting): authenticate will time out at 1255.034547s 2023-09-30T01:56:05.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98ea4d8700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f98e4096a50 0x7f98e4132680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42600/0 (socket says 172.21.15.7:42600) 2023-09-30T01:56:05.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98e9cd7700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f98e4096e20 0x7f98e40971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:05.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98ea4d8700 1 -- 172.21.15.7:0/3660381054 learned_addr learned my addr 172.21.15.7:0/3660381054 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:56:05.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98e9cd7700 10 monclient(hunting): get_auth_request con 0x7f98e4096e20 auth_method 0 2023-09-30T01:56:05.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98e9cd7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:05.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98e9cd7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:05.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98e9cd7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:05.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98e9cd7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:05.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98e9cd7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:05.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98e9cd7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:05.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98eacd9700 1 --2- 172.21.15.7:0/3660381054 >> v2:172.21.15.50:3300/0 conn(0x7f98e4137710 0x7f98e4139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:05.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98eacd9700 10 monclient(hunting): get_auth_request con 0x7f98e4137710 auth_method 0 2023-09-30T01:56:05.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98eacd9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:05.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98eacd9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:05.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98eacd9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:05.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98e9cd7700 10 monclient(hunting): handle_auth_done global_id 4421 payload 274 2023-09-30T01:56:05.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98e9cd7700 1 -- 172.21.15.7:0/3660381054 >> v2:172.21.15.50:3300/0 conn(0x7f98e4137710 msgr2=0x7f98e4139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.969+0000 7f98e9cd7700 1 --2- 172.21.15.7:0/3660381054 >> v2:172.21.15.50:3300/0 conn(0x7f98e4137710 0x7f98e4139af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.970+0000 7f98e9cd7700 1 -- 172.21.15.7:0/3660381054 >> v1:172.21.15.7:6789/0 conn(0x7f98e4096a50 legacy=0x7f98e4132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.970+0000 7f98e9cd7700 10 monclient: _finish_hunting 0 2023-09-30T01:56:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.970+0000 7f98e9cd7700 1 monclient: found mon.noname-b 2023-09-30T01:56:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.970+0000 7f98e9cd7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.970+0000 7f98e9cd7700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:56:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.970+0000 7f98e9cd7700 1 -- 172.21.15.7:0/3660381054 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e4099720 con 0x7f98e4096e20 2023-09-30T01:56:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98e9cd7700 1 -- 172.21.15.7:0/3660381054 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f98e4096e20 msgr2=0x7f98e40971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98e9cd7700 1 --2- 172.21.15.7:0/3660381054 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f98e4096e20 0x7f98e40971f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f98d80098e0 tx=0x7f98d80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad9bd9fd6f9c7b0c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98e94d6700 1 -- 172.21.15.7:0/3660381054 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d8009e80 con 0x7f98e4096e20 2023-09-30T01:56:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98e94d6700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98e94d6700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:56:05.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98e94d6700 10 monclient: dump: 2023-09-30T01:56:05.977 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:05.977 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:05.977 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:05.977 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:05.977 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:05.977 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:05.977 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:05.977 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:05.977 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:05.977 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:05.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98e94d6700 10 monclient: _finish_auth 0 2023-09-30T01:56:05.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98e94d6700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:05.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98ec73c700 5 monclient: authenticate success, global_id 4421 2023-09-30T01:56:05.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98e94d6700 1 -- 172.21.15.7:0/3660381054 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d800ba30 con 0x7f98e4096e20 2023-09-30T01:56:05.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98ec73c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:56:05.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98e94d6700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:05.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98ec73c700 10 monclient: get_monmap_and_config success 2023-09-30T01:56:05.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98ec73c700 4 set_mon_vals no callback set 2023-09-30T01:56:05.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98e94d6700 1 -- 172.21.15.7:0/3660381054 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d800bdb0 con 0x7f98e4096e20 2023-09-30T01:56:05.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98e94d6700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:05.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98e94d6700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:56:05.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98e94d6700 10 monclient: dump: 2023-09-30T01:56:05.978 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:05.979 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:05.979 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:05.979 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:05.979 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:05.979 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:05.979 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:05.979 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:05.979 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:05.979 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:05.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98ec73c700 10 monclient: shutdown 2023-09-30T01:56:05.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98ec73c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:05.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98ec73c700 1 -- 172.21.15.7:0/3660381054 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f98e4096e20 msgr2=0x7f98e40971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:05.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.972+0000 7f98ec73c700 1 --2- 172.21.15.7:0/3660381054 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f98e4096e20 0x7f98e40971f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f98d80098e0 tx=0x7f98d80091a0 comp rx=0 tx=0).stop 2023-09-30T01:56:05.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.973+0000 7f98ec73c700 1 -- 172.21.15.7:0/3660381054 shutdown_connections 2023-09-30T01:56:05.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.973+0000 7f98ec73c700 1 --2- 172.21.15.7:0/3660381054 >> v2:172.21.15.50:3300/0 conn(0x7f98e4137710 0x7f98e4139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:05.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.973+0000 7f98ec73c700 1 --2- 172.21.15.7:0/3660381054 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f98e4096e20 0x7f98e40971f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:05.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.973+0000 7f98ec73c700 1 -- 172.21.15.7:0/3660381054 >> 172.21.15.7:0/3660381054 conn(0x7f98e4070cf0 msgr2=0x7f98e40710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:05.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.973+0000 7f98ec73c700 1 -- 172.21.15.7:0/3660381054 shutdown_connections 2023-09-30T01:56:05.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.973+0000 7f98ec73c700 1 -- 172.21.15.7:0/3660381054 wait complete. 2023-09-30T01:56:05.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.973+0000 7f98ec73c700 10 monclient: build_initial_monmap 2023-09-30T01:56:05.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.973+0000 7f98ec73c700 10 monclient: monmap: 2023-09-30T01:56:05.984 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:05.985 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:56:05.985 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:56:05.985 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:56:05.985 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:05.985 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:05.985 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:56:05.985 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:56:05.985 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:56:05.985 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:05.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.973+0000 7f98ec73c700 1 Processor -- start 2023-09-30T01:56:05.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ec73c700 1 -- start start 2023-09-30T01:56:05.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ec73c700 10 monclient: init 2023-09-30T01:56:05.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ec73c700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:05.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ec73c700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:56:05.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ec73c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98e4096a50 0x7f98e40a1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:05.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ec73c700 10 monclient(hunting): picked mon.noname-b con 0x7f98e4096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:56:05.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ec73c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98e4096e20 0x7f98e409cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:05.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ec73c700 10 monclient(hunting): picked mon.noname-a con 0x7f98e4096e20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:56:05.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ec73c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f98e4137710 0x7f98e409d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:05.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ec73c700 10 monclient(hunting): picked mon.noname-c con 0x7f98e4137710 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:05.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ec73c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:05.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ec73c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e41334e0 con 0x7f98e4096e20 2023-09-30T01:56:05.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ec73c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:05.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ec73c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e4131df0 con 0x7f98e4096a50 2023-09-30T01:56:05.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ec73c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:05.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ec73c700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40a0340 con 0x7f98e4137710 2023-09-30T01:56:05.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ec73c700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:05.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ec73c700 10 monclient(hunting): authenticate will time out at 1255.039505s 2023-09-30T01:56:05.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98eacd9700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f98e4137710 0x7f98e409d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:05.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98e9cd7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98e4096e20 0x7f98e409cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:05.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98eacd9700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f98e4137710 0x7f98e409d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:56774/0 (socket says 172.21.15.7:56774) 2023-09-30T01:56:05.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98eacd9700 1 -- 172.21.15.7:0/853515984 learned_addr learned my addr 172.21.15.7:0/853515984 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:05.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98e9cd7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98e4096e20 0x7f98e409cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47378/0 (socket says 172.21.15.7:47378) 2023-09-30T01:56:05.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98eacd9700 10 monclient(hunting): get_auth_request con 0x7f98e4137710 auth_method 0 2023-09-30T01:56:05.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98eacd9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:05.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98eacd9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:05.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98eacd9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:05.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98e9cd7700 10 monclient(hunting): get_auth_request con 0x7f98e4096e20 auth_method 0 2023-09-30T01:56:05.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98e9cd7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:05.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98e9cd7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:05.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98e9cd7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:05.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98eacd9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:05.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98eacd9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:05.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ea4d8700 1 --2- 172.21.15.7:0/853515984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98e4096a50 0x7f98e40a1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:05.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98eacd9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:05.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98e9cd7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:05.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98e9cd7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:05.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98e9cd7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:05.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ea4d8700 10 monclient(hunting): get_auth_request con 0x7f98e4096a50 auth_method 0 2023-09-30T01:56:05.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ea4d8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:05.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ea4d8700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:05.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ea4d8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:05.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ea4d8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:05.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ea4d8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:05.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98ea4d8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:05.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98eacd9700 10 monclient(hunting): handle_auth_done global_id 4424 payload 824 2023-09-30T01:56:05.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98eacd9700 1 -- 172.21.15.7:0/853515984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98e4096a50 msgr2=0x7f98e40a1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:05.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98eacd9700 1 --2- 172.21.15.7:0/853515984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98e4096a50 0x7f98e40a1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:05.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98eacd9700 1 -- 172.21.15.7:0/853515984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98e4096e20 msgr2=0x7f98e409cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:05.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98eacd9700 1 --2- 172.21.15.7:0/853515984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98e4096e20 0x7f98e409cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:05.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98eacd9700 10 monclient: _finish_hunting 0 2023-09-30T01:56:05.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98eacd9700 1 monclient: found mon.noname-c 2023-09-30T01:56:05.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98eacd9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:05.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98eacd9700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:56:05.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98eacd9700 1 -- 172.21.15.7:0/853515984 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d8009420 con 0x7f98e4137710 2023-09-30T01:56:05.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.974+0000 7f98e9cd7700 1 --2- 172.21.15.7:0/853515984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98e4096e20 0x7f98e409cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:56:05.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98eacd9700 1 -- 172.21.15.7:0/853515984 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f98e4137710 msgr2=0x7f98e409d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:05.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98eacd9700 1 --2- 172.21.15.7:0/853515984 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f98e4137710 0x7f98e409d280 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f98d400d860 tx=0x7f98d4008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:05.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98e27fc700 1 -- 172.21.15.7:0/853515984 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d4012800 con 0x7f98e4137710 2023-09-30T01:56:05.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:05.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98e27fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:56:05.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98e27fc700 10 monclient: dump: 2023-09-30T01:56:05.990 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:05.990 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:05.991 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:05.991 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:05.991 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:05.991 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:05.991 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:05.991 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:05.991 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:05.991 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:05.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98e27fc700 10 monclient: _finish_auth 0 2023-09-30T01:56:05.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98e27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:05.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98e27fc700 1 -- 172.21.15.7:0/853515984 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d4012960 con 0x7f98e4137710 2023-09-30T01:56:05.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98ec73c700 5 monclient: authenticate success, global_id 4424 2023-09-30T01:56:05.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98ec73c700 10 monclient: _renew_subs 2023-09-30T01:56:05.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98ec73c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:05.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98ec73c700 1 -- 172.21.15.7:0/853515984 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e409e810 con 0x7f98e4137710 2023-09-30T01:56:05.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98e27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:05.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98e27fc700 1 -- 172.21.15.7:0/853515984 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d4012d20 con 0x7f98e4137710 2023-09-30T01:56:05.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:05.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98e27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:56:05.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98e27fc700 10 monclient: dump: 2023-09-30T01:56:05.992 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:05.992 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:05.992 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:05.992 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:05.992 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:05.993 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98e8cd5700 4 set_mon_vals no callback set 2023-09-30T01:56:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98ec73c700 10 monclient: _renew_subs 2023-09-30T01:56:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98ec73c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.977+0000 7f98ec73c700 1 -- 172.21.15.7:0/853515984 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e409d7c0 con 0x7f98e4137710 2023-09-30T01:56:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.978+0000 7f98ec73c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.978+0000 7f98ec73c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.978+0000 7f98ec73c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.978+0000 7f98ec73c700 1 -- 172.21.15.7:0/853515984 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e4064c10 con 0x7f98e4137710 2023-09-30T01:56:05.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.978+0000 7f98e27fc700 1 -- 172.21.15.7:0/853515984 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f98d4013690 con 0x7f98e4137710 2023-09-30T01:56:05.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.978+0000 7f98e27fc700 1 --2- 172.21.15.7:0/853515984 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f98d00357e0 0x7f98d0037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:05.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.978+0000 7f98e27fc700 1 -- 172.21.15.7:0/853515984 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f98d40480a0 con 0x7f98e4137710 2023-09-30T01:56:05.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.978+0000 7f98ea4d8700 1 --2- 172.21.15.7:0/853515984 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f98d00357e0 0x7f98d0037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:05.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.978+0000 7f98ea4d8700 10 monclient: get_auth_request con 0x7f98d00357e0 auth_method 0 2023-09-30T01:56:05.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.981+0000 7f98ea4d8700 1 --2- 172.21.15.7:0/853515984 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f98d00357e0 0x7f98d0037c90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f98dc006e80 tx=0x7f98dc0053c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:05.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.983+0000 7f98e27fc700 1 -- 172.21.15.7:0/853515984 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f98d4013940 con 0x7f98e4137710 2023-09-30T01:56:05.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.983+0000 7f98e27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:05.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:05.983+0000 7f98e27fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:56:06.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:06.218+0000 7f98ec73c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:56:06.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:06.218+0000 7f98ec73c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:06.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:06.218+0000 7f98ec73c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:06.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:06.218+0000 7f98ec73c700 1 -- 172.21.15.7:0/853515984 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f98e4097730 con 0x7f98e4137710 2023-09-30T01:56:06.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:06.218+0000 7f98e27fc700 1 -- 172.21.15.7:0/853515984 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f98d400f310 con 0x7f98e4137710 2023-09-30T01:56:06.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:06.218+0000 7f98e27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:06.219 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:06.218+0000 7f98e27fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:56:06.219 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:56:06.220 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:56:06.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:06.220+0000 7f98ec73c700 1 -- 172.21.15.7:0/853515984 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f98d00357e0 msgr2=0x7f98d0037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:06.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:06.220+0000 7f98ec73c700 1 --2- 172.21.15.7:0/853515984 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f98d00357e0 0x7f98d0037c90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f98dc006e80 tx=0x7f98dc0053c0 comp rx=0 tx=0).stop 2023-09-30T01:56:06.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:06.220+0000 7f98ec73c700 10 monclient: shutdown 2023-09-30T01:56:06.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:06.220+0000 7f98ec73c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:06.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:06.220+0000 7f98ec73c700 1 -- 172.21.15.7:0/853515984 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f98e4137710 msgr2=0x7f98e409d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:06.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:06.220+0000 7f98ec73c700 1 --2- 172.21.15.7:0/853515984 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f98e4137710 0x7f98e409d280 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f98d400d860 tx=0x7f98d4008040 comp rx=0 tx=0).stop 2023-09-30T01:56:06.221 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:06.220+0000 7f98ec73c700 1 -- 172.21.15.7:0/853515984 shutdown_connections 2023-09-30T01:56:06.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:06.220+0000 7f98ec73c700 1 --2- 172.21.15.7:0/853515984 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f98d00357e0 0x7f98d0037c90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:06.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:06.220+0000 7f98ec73c700 1 --2- 172.21.15.7:0/853515984 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f98e4137710 0x7f98e409d280 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:06.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:06.220+0000 7f98ec73c700 1 --2- 172.21.15.7:0/853515984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98e4096a50 0x7f98e40a1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:06.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:06.220+0000 7f98ec73c700 1 --2- 172.21.15.7:0/853515984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f98e4096e20 0x7f98e409cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:06.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:06.221+0000 7f98ec73c700 1 -- 172.21.15.7:0/853515984 >> 172.21.15.7:0/853515984 conn(0x7f98e4070cf0 msgr2=0x7f98e40a4500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:06.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:06.221+0000 7f98ec73c700 1 -- 172.21.15.7:0/853515984 shutdown_connections 2023-09-30T01:56:06.222 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:06.221+0000 7f98ec73c700 1 -- 172.21.15.7:0/853515984 wait complete. 2023-09-30T01:56:09.234 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:56:09.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.364+0000 7fd871017700 10 monclient: get_monmap_and_config 2023-09-30T01:56:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.364+0000 7fd871017700 10 monclient: build_initial_monmap 2023-09-30T01:56:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.364+0000 7fd871017700 10 monclient: monmap: 2023-09-30T01:56:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:56:09.365955+0000 2023-09-30T01:56:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:56:09.365955+0000 2023-09-30T01:56:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:09.365 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:56:09.366 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:56:09.366 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:56:09.366 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:56:09.366 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:56:09.366 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.365+0000 7fd871017700 1 Processor -- start 2023-09-30T01:56:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.365+0000 7fd871017700 1 -- start start 2023-09-30T01:56:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.365+0000 7fd871017700 10 monclient: init 2023-09-30T01:56:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd871017700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd871017700 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-09-30T01:56:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd871017700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fd86c098540 0x7fd86c1329e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:09.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd871017700 10 monclient(hunting): picked mon.noname-c con 0x7fd86c098540 addr v2:172.21.15.50:3300/0 2023-09-30T01:56:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd871017700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd86c098910 0x7fd86c098d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd871017700 10 monclient(hunting): picked mon.noname-a con 0x7fd86c098910 addr v2:172.21.15.7:3300/0 2023-09-30T01:56:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd871017700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd86c12ebc0 0x7fd86c12efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd871017700 10 monclient(hunting): picked mon.noname-b con 0x7fd86c12ebc0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd871017700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd871017700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd86c0a0f40 con 0x7fd86c098910 2023-09-30T01:56:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd871017700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd871017700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd86c069f10 con 0x7fd86c098540 2023-09-30T01:56:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd871017700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd871017700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c0a05c0 con 0x7fd86c12ebc0 2023-09-30T01:56:09.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd871017700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:09.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd871017700 10 monclient(hunting): authenticate will time out at 1258.431203s 2023-09-30T01:56:09.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd86b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd86c098910 0x7fd86c098d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:09.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd870816700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd86c12ebc0 0x7fd86c12efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:09.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd870816700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd86c12ebc0 0x7fd86c12efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:56782/0 (socket says 172.21.15.7:56782) 2023-09-30T01:56:09.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd86b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fd86c098910 0x7fd86c098d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47384/0 (socket says 172.21.15.7:47384) 2023-09-30T01:56:09.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd870816700 1 -- 172.21.15.7:0/624082934 learned_addr learned my addr 172.21.15.7:0/624082934 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:09.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd870816700 10 monclient(hunting): get_auth_request con 0x7fd86c12ebc0 auth_method 0 2023-09-30T01:56:09.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd870816700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:09.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd870816700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:09.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd870816700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:09.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd86b7fe700 10 monclient(hunting): get_auth_request con 0x7fd86c098910 auth_method 0 2023-09-30T01:56:09.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd86b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:09.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd86b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:09.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd86b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:09.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd86b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:09.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd86b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:09.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd86b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:09.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd870816700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:09.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd870816700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:09.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd870816700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:09.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd86bfff700 1 --2- 172.21.15.7:0/624082934 >> v2:172.21.15.50:3300/0 conn(0x7fd86c098540 0x7fd86c1329e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:09.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd86bfff700 10 monclient(hunting): get_auth_request con 0x7fd86c098540 auth_method 0 2023-09-30T01:56:09.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd86bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:09.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd86bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd86bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd870816700 10 monclient(hunting): handle_auth_done global_id 4427 payload 274 2023-09-30T01:56:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd870816700 1 -- 172.21.15.7:0/624082934 >> v2:172.21.15.50:3300/0 conn(0x7fd86c098540 msgr2=0x7fd86c1329e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd870816700 1 --2- 172.21.15.7:0/624082934 >> v2:172.21.15.50:3300/0 conn(0x7fd86c098540 0x7fd86c1329e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd870816700 1 -- 172.21.15.7:0/624082934 >> v2:172.21.15.7:3300/0 conn(0x7fd86c098910 msgr2=0x7fd86c098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd870816700 1 --2- 172.21.15.7:0/624082934 >> v2:172.21.15.7:3300/0 conn(0x7fd86c098910 0x7fd86c098d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd870816700 10 monclient: _finish_hunting 0 2023-09-30T01:56:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd870816700 1 monclient: found mon.noname-b 2023-09-30T01:56:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd870816700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd870816700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:56:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd870816700 1 -- 172.21.15.7:0/624082934 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c096a50 con 0x7fd86c12ebc0 2023-09-30T01:56:09.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.366+0000 7fd86b7fe700 1 --2- 172.21.15.7:0/624082934 >> v2:172.21.15.7:3300/0 conn(0x7fd86c098910 0x7fd86c098d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:56:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.368+0000 7fd870816700 1 -- 172.21.15.7:0/624082934 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd86c12ebc0 msgr2=0x7fd86c12efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.368+0000 7fd870816700 1 --2- 172.21.15.7:0/624082934 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd86c12ebc0 0x7fd86c12efe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd86c071cb0 tx=0x7fd864009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9be985dd91070d2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd86affd700 1 -- 172.21.15.7:0/624082934 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86400e070 con 0x7fd86c12ebc0 2023-09-30T01:56:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd86affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd86affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:56:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd86affd700 10 monclient: dump: 2023-09-30T01:56:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:09.378 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:09.379 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd86affd700 10 monclient: _finish_auth 0 2023-09-30T01:56:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd86affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd871017700 5 monclient: authenticate success, global_id 4427 2023-09-30T01:56:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd86affd700 1 -- 172.21.15.7:0/624082934 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86400bba0 con 0x7fd86c12ebc0 2023-09-30T01:56:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd871017700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:56:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd86affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:09.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd871017700 10 monclient: get_monmap_and_config success 2023-09-30T01:56:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd871017700 4 set_mon_vals no callback set 2023-09-30T01:56:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd86affd700 1 -- 172.21.15.7:0/624082934 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8640092e0 con 0x7fd86c12ebc0 2023-09-30T01:56:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd86affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd86affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:56:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd86affd700 10 monclient: dump: 2023-09-30T01:56:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:09.381 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:09.381 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:09.381 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:09.381 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:09.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd871017700 10 monclient: shutdown 2023-09-30T01:56:09.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd871017700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:09.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd871017700 1 -- 172.21.15.7:0/624082934 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd86c12ebc0 msgr2=0x7fd86c12efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:09.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd871017700 1 --2- 172.21.15.7:0/624082934 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd86c12ebc0 0x7fd86c12efe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd86c071cb0 tx=0x7fd864009840 comp rx=0 tx=0).stop 2023-09-30T01:56:09.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd871017700 1 -- 172.21.15.7:0/624082934 shutdown_connections 2023-09-30T01:56:09.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd871017700 1 --2- 172.21.15.7:0/624082934 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd86c12ebc0 0x7fd86c12efe0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:09.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd871017700 1 --2- 172.21.15.7:0/624082934 >> v2:172.21.15.50:3300/0 conn(0x7fd86c098540 0x7fd86c1329e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:09.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd871017700 1 --2- 172.21.15.7:0/624082934 >> v2:172.21.15.7:3300/0 conn(0x7fd86c098910 0x7fd86c098d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:09.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd871017700 1 -- 172.21.15.7:0/624082934 >> 172.21.15.7:0/624082934 conn(0x7fd86c071830 msgr2=0x7fd86c0a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:09.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.369+0000 7fd871017700 1 -- 172.21.15.7:0/624082934 shutdown_connections 2023-09-30T01:56:09.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.370+0000 7fd871017700 1 -- 172.21.15.7:0/624082934 wait complete. 2023-09-30T01:56:09.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.370+0000 7fd871017700 10 monclient: build_initial_monmap 2023-09-30T01:56:09.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.370+0000 7fd871017700 10 monclient: monmap: 2023-09-30T01:56:09.382 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:09.382 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:56:09.382 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:56:09.382 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:56:09.382 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:09.382 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:09.382 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:56:09.383 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:56:09.383 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:56:09.383 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:09.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.370+0000 7fd871017700 1 Processor -- start 2023-09-30T01:56:09.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd871017700 1 -- start start 2023-09-30T01:56:09.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd871017700 10 monclient: init 2023-09-30T01:56:09.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd871017700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:09.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd871017700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:56:09.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd871017700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c098540 0x7fd86c133620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:09.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd871017700 10 monclient(hunting): picked mon.noname-a con 0x7fd86c098540 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:56:09.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd871017700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd86c098910 0x7fd86c09b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:09.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd871017700 10 monclient(hunting): picked mon.noname-b con 0x7fd86c098910 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:56:09.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd871017700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd86c12ebc0 0x7fd86c12dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:09.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd871017700 10 monclient(hunting): picked mon.noname-c con 0x7fd86c12ebc0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:09.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd871017700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:09.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd871017700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c11e4a0 con 0x7fd86c098540 2023-09-30T01:56:09.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd871017700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:09.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd871017700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c06acf0 con 0x7fd86c098910 2023-09-30T01:56:09.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd871017700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:09.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd871017700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c1420f0 con 0x7fd86c12ebc0 2023-09-30T01:56:09.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd871017700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:09.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd871017700 10 monclient(hunting): authenticate will time out at 1258.436482s 2023-09-30T01:56:09.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd870816700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd86c12ebc0 0x7fd86c12dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:09.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd86bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c098540 0x7fd86c133620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:09.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd870816700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd86c12ebc0 0x7fd86c12dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:56798/0 (socket says 172.21.15.7:56798) 2023-09-30T01:56:09.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd870816700 1 -- 172.21.15.7:0/1810365546 learned_addr learned my addr 172.21.15.7:0/1810365546 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:09.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd870816700 10 monclient(hunting): get_auth_request con 0x7fd86c12ebc0 auth_method 0 2023-09-30T01:56:09.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd870816700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:09.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd870816700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:09.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd870816700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:09.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd86bfff700 10 monclient(hunting): get_auth_request con 0x7fd86c098540 auth_method 0 2023-09-30T01:56:09.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd86bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:09.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd86bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:09.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd86bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:09.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd86b7fe700 1 --2- 172.21.15.7:0/1810365546 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd86c098910 0x7fd86c09b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:09.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd870816700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:09.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd870816700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:09.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd870816700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:09.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd86b7fe700 10 monclient(hunting): get_auth_request con 0x7fd86c098910 auth_method 0 2023-09-30T01:56:09.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd86b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:09.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd86b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:09.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd86b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:09.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd86bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:09.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd86bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:09.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd86bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:09.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd86b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:09.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd86b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:09.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd86b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:09.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd870816700 10 monclient(hunting): handle_auth_done global_id 4430 payload 824 2023-09-30T01:56:09.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd870816700 1 -- 172.21.15.7:0/1810365546 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd86c098910 msgr2=0x7fd86c09b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:09.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd870816700 1 --2- 172.21.15.7:0/1810365546 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd86c098910 0x7fd86c09b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:09.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd870816700 1 -- 172.21.15.7:0/1810365546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c098540 msgr2=0x7fd86c133620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:09.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd870816700 1 --2- 172.21.15.7:0/1810365546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c098540 0x7fd86c133620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:09.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd870816700 10 monclient: _finish_hunting 0 2023-09-30T01:56:09.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd870816700 1 monclient: found mon.noname-c 2023-09-30T01:56:09.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd870816700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:09.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd870816700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:56:09.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.371+0000 7fd870816700 1 -- 172.21.15.7:0/1810365546 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85c009420 con 0x7fd86c12ebc0 2023-09-30T01:56:09.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.372+0000 7fd86bfff700 1 --2- 172.21.15.7:0/1810365546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c098540 0x7fd86c133620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:56:09.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd870816700 1 -- 172.21.15.7:0/1810365546 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd86c12ebc0 msgr2=0x7fd86c12dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:09.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd870816700 1 --2- 172.21.15.7:0/1810365546 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd86c12ebc0 0x7fd86c12dd00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd8640032c0 tx=0x7fd864018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:09.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd853fff700 1 -- 172.21.15.7:0/1810365546 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd864007190 con 0x7fd86c12ebc0 2023-09-30T01:56:09.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd853fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:09.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd853fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:56:09.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd853fff700 10 monclient: dump: 2023-09-30T01:56:09.390 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:09.390 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:09.390 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:09.391 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:09.391 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:09.391 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:09.391 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:09.391 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:09.391 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:09.391 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:09.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd853fff700 10 monclient: _finish_auth 0 2023-09-30T01:56:09.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd853fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:09.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd853fff700 1 -- 172.21.15.7:0/1810365546 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd864007ab0 con 0x7fd86c12ebc0 2023-09-30T01:56:09.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd871017700 5 monclient: authenticate success, global_id 4430 2023-09-30T01:56:09.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd871017700 10 monclient: _renew_subs 2023-09-30T01:56:09.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd871017700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:09.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd871017700 1 -- 172.21.15.7:0/1810365546 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd864009ae0 con 0x7fd86c12ebc0 2023-09-30T01:56:09.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd853fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:09.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd853fff700 1 -- 172.21.15.7:0/1810365546 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd864007e90 con 0x7fd86c12ebc0 2023-09-30T01:56:09.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd853fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:09.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd853fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:56:09.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd853fff700 10 monclient: dump: 2023-09-30T01:56:09.392 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:09.392 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:09.392 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:09.392 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:09.393 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:09.393 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:09.393 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:09.393 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:09.393 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:09.393 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:09.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd871017700 10 monclient: _renew_subs 2023-09-30T01:56:09.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd871017700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:09.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd871017700 1 -- 172.21.15.7:0/1810365546 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd86c11cf60 con 0x7fd86c12ebc0 2023-09-30T01:56:09.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.374+0000 7fd86a7fc700 4 set_mon_vals no callback set 2023-09-30T01:56:09.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.375+0000 7fd870816700 1 --2- 172.21.15.7:0/1810365546 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd86c12ebc0 0x7fd86c12dd00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd8640032c0 tx=0x7fd864018000 comp rx=0 tx=0).queue_received will delay after 1.69604e+09 on 0x7fd864026670 mgrmap(e 5) v1 2023-09-30T01:56:09.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.375+0000 7fd853fff700 1 -- 172.21.15.7:0/1810365546 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fd864026670 con 0x7fd86c12ebc0 2023-09-30T01:56:09.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.375+0000 7fd871017700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:09.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.375+0000 7fd871017700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:09.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.375+0000 7fd871017700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:09.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.375+0000 7fd871017700 1 -- 172.21.15.7:0/1810365546 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd86c09bd60 con 0x7fd86c12ebc0 2023-09-30T01:56:09.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.375+0000 7fd853fff700 1 --2- 172.21.15.7:0/1810365546 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fd8580357e0 0x7fd858037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:09.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.375+0000 7fd86bfff700 1 --2- 172.21.15.7:0/1810365546 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fd8580357e0 0x7fd858037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:09.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.375+0000 7fd86bfff700 10 monclient: get_auth_request con 0x7fd8580357e0 auth_method 0 2023-09-30T01:56:09.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.378+0000 7fd86bfff700 1 --2- 172.21.15.7:0/1810365546 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fd8580357e0 0x7fd858037c90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd860006fd0 tx=0x7fd860008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:09.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:09.380+0000 7fd853fff700 1 -- 172.21.15.7:0/1810365546 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fd864049a50 con 0x7fd86c12ebc0 2023-09-30T01:56:10.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.203+0000 7fd853fff700 1 -- 172.21.15.7:0/1810365546 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd86401d420 con 0x7fd86c12ebc0 2023-09-30T01:56:10.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.203+0000 7fd853fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:10.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.203+0000 7fd853fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:56:10.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.441+0000 7fd871017700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:56:10.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.441+0000 7fd871017700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:10.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.441+0000 7fd871017700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:10.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.441+0000 7fd871017700 1 -- 172.21.15.7:0/1810365546 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd86c064c10 con 0x7fd86c12ebc0 2023-09-30T01:56:10.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.442+0000 7fd853fff700 1 -- 172.21.15.7:0/1810365546 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd86402b020 con 0x7fd86c12ebc0 2023-09-30T01:56:10.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.442+0000 7fd853fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:10.442 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.442+0000 7fd853fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:56:10.443 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:56:10.443 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:56:10.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.443+0000 7fd871017700 1 -- 172.21.15.7:0/1810365546 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fd8580357e0 msgr2=0x7fd858037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:10.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.443+0000 7fd871017700 1 --2- 172.21.15.7:0/1810365546 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fd8580357e0 0x7fd858037c90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd860006fd0 tx=0x7fd860008040 comp rx=0 tx=0).stop 2023-09-30T01:56:10.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.443+0000 7fd871017700 10 monclient: shutdown 2023-09-30T01:56:10.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.443+0000 7fd871017700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:10.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.443+0000 7fd871017700 1 -- 172.21.15.7:0/1810365546 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd86c12ebc0 msgr2=0x7fd86c12dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:10.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.443+0000 7fd871017700 1 --2- 172.21.15.7:0/1810365546 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd86c12ebc0 0x7fd86c12dd00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd8640032c0 tx=0x7fd864018000 comp rx=0 tx=0).stop 2023-09-30T01:56:10.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.444+0000 7fd871017700 1 -- 172.21.15.7:0/1810365546 shutdown_connections 2023-09-30T01:56:10.444 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.444+0000 7fd871017700 1 --2- 172.21.15.7:0/1810365546 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fd8580357e0 0x7fd858037c90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:10.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.444+0000 7fd871017700 1 --2- 172.21.15.7:0/1810365546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd86c098540 0x7fd86c133620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:10.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.444+0000 7fd871017700 1 --2- 172.21.15.7:0/1810365546 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd86c12ebc0 0x7fd86c12dd00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:10.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.444+0000 7fd871017700 1 --2- 172.21.15.7:0/1810365546 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd86c098910 0x7fd86c09b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:10.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.444+0000 7fd871017700 1 -- 172.21.15.7:0/1810365546 >> 172.21.15.7:0/1810365546 conn(0x7fd86c071830 msgr2=0x7fd86c0a39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:10.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.444+0000 7fd871017700 1 -- 172.21.15.7:0/1810365546 shutdown_connections 2023-09-30T01:56:10.445 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:10.444+0000 7fd871017700 1 -- 172.21.15.7:0/1810365546 wait complete. 2023-09-30T01:56:13.456 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:56:13.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.586+0000 7fe7fa433700 10 monclient: get_monmap_and_config 2023-09-30T01:56:13.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.586+0000 7fe7fa433700 10 monclient: build_initial_monmap 2023-09-30T01:56:13.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.586+0000 7fe7fa433700 10 monclient: monmap: 2023-09-30T01:56:13.587 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:13.587 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:13.587 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:56:13.587827+0000 2023-09-30T01:56:13.587 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:56:13.587827+0000 2023-09-30T01:56:13.587 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:13.587 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:13.588 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:56:13.588 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:56:13.588 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:56:13.588 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:56:13.588 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:56:13.588 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:13.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.587+0000 7fe7fa433700 1 Processor -- start 2023-09-30T01:56:13.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.587+0000 7fe7fa433700 1 -- start start 2023-09-30T01:56:13.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.587+0000 7fe7fa433700 10 monclient: init 2023-09-30T01:56:13.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7fa433700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:13.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7fa433700 10 monclient: _add_conns ranks=[4,3,0,1,2] 2023-09-30T01:56:13.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7fa433700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fe7f4098540 0x7fe7f41329e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:13.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7fa433700 10 monclient(hunting): picked mon.noname-c con 0x7fe7f4098540 addr v2:172.21.15.50:3300/0 2023-09-30T01:56:13.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7fa433700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe7f4098910 0x7fe7f4098d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:13.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7fa433700 10 monclient(hunting): picked mon.noname-a con 0x7fe7f4098910 addr v2:172.21.15.7:3300/0 2023-09-30T01:56:13.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7fa433700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe7f412ebc0 addr v1:172.21.15.7:6789/0 2023-09-30T01:56:13.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7fa433700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f4069f10 con 0x7fe7f412ebc0 2023-09-30T01:56:13.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7fa433700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:13.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7fa433700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f40a0f40 con 0x7fe7f4098910 2023-09-30T01:56:13.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7fa433700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:13.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7fa433700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f4132f20 con 0x7fe7f4098540 2023-09-30T01:56:13.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7fa433700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:13.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7fa433700 10 monclient(hunting): authenticate will time out at 1262.653224s 2023-09-30T01:56:13.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f8c30700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fe7f4098910 0x7fe7f4098d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:13.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f9c32700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe7f412ebc0 0x7fe7f412efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42606/0 (socket says 172.21.15.7:42606) 2023-09-30T01:56:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f9c32700 1 -- 172.21.15.7:0/4152724483 learned_addr learned my addr 172.21.15.7:0/4152724483 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:56:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f8c30700 10 monclient(hunting): get_auth_request con 0x7fe7f4098910 auth_method 0 2023-09-30T01:56:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f8c30700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f8c30700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f8c30700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f8c30700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f8c30700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f8c30700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f9431700 1 --2- 172.21.15.7:0/4152724483 >> v2:172.21.15.50:3300/0 conn(0x7fe7f4098540 0x7fe7f41329e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f9431700 10 monclient(hunting): get_auth_request con 0x7fe7f4098540 auth_method 0 2023-09-30T01:56:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f9431700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:13.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f9431700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:13.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f9431700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:13.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f8c30700 10 monclient(hunting): handle_auth_done global_id 4419 payload 274 2023-09-30T01:56:13.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f8c30700 1 -- 172.21.15.7:0/4152724483 >> v2:172.21.15.50:3300/0 conn(0x7fe7f4098540 msgr2=0x7fe7f41329e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:13.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f8c30700 1 --2- 172.21.15.7:0/4152724483 >> v2:172.21.15.50:3300/0 conn(0x7fe7f4098540 0x7fe7f41329e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:13.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f8c30700 1 -- 172.21.15.7:0/4152724483 >> v1:172.21.15.7:6789/0 conn(0x7fe7f412ebc0 legacy=0x7fe7f412efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:13.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f8c30700 10 monclient: _finish_hunting 0 2023-09-30T01:56:13.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f8c30700 1 monclient: found mon.noname-a 2023-09-30T01:56:13.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f8c30700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:13.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f8c30700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:56:13.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.588+0000 7fe7f8c30700 1 -- 172.21.15.7:0/4152724483 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f4096a50 con 0x7fe7f4098910 2023-09-30T01:56:13.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7f8c30700 1 -- 172.21.15.7:0/4152724483 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f4098910 msgr2=0x7fe7f4098d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:13.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7f8c30700 1 --2- 172.21.15.7:0/4152724483 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f4098910 0x7fe7f4098d10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4071cb0 tx=0x7fe7e80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bc56c670d8b6f81 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:13.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7f3fff700 1 -- 172.21.15.7:0/4152724483 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e800b880 con 0x7fe7f4098910 2023-09-30T01:56:13.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7f3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:13.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7f3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:56:13.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7f3fff700 10 monclient: dump: 2023-09-30T01:56:13.597 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:13.597 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:13.597 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:13.597 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:13.597 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:13.597 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:13.597 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:13.598 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:13.598 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:13.598 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:13.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7f3fff700 10 monclient: _finish_auth 0 2023-09-30T01:56:13.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7f3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:13.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7f3fff700 1 -- 172.21.15.7:0/4152724483 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e800b9e0 con 0x7fe7f4098910 2023-09-30T01:56:13.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7fa433700 5 monclient: authenticate success, global_id 4419 2023-09-30T01:56:13.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7fa433700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:56:13.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7f3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:13.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7f3fff700 1 -- 172.21.15.7:0/4152724483 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e800bd80 con 0x7fe7f4098910 2023-09-30T01:56:13.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7fa433700 10 monclient: get_monmap_and_config success 2023-09-30T01:56:13.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7fa433700 4 set_mon_vals no callback set 2023-09-30T01:56:13.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7f3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:13.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7f3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:56:13.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7f3fff700 10 monclient: dump: 2023-09-30T01:56:13.599 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:13.599 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:13.599 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:13.599 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:13.599 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:13.599 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:13.599 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:13.599 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:13.599 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:13.600 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:13.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7fa433700 10 monclient: shutdown 2023-09-30T01:56:13.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7fa433700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:13.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7fa433700 1 -- 172.21.15.7:0/4152724483 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f4098910 msgr2=0x7fe7f4098d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:13.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7fa433700 1 --2- 172.21.15.7:0/4152724483 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f4098910 0x7fe7f4098d10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4071cb0 tx=0x7fe7e80091a0 comp rx=0 tx=0).stop 2023-09-30T01:56:13.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7fa433700 1 -- 172.21.15.7:0/4152724483 shutdown_connections 2023-09-30T01:56:13.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7fa433700 1 --2- 172.21.15.7:0/4152724483 >> v2:172.21.15.50:3300/0 conn(0x7fe7f4098540 0x7fe7f41329e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:13.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7fa433700 1 --2- 172.21.15.7:0/4152724483 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f4098910 0x7fe7f4098d10 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:13.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7fa433700 1 -- 172.21.15.7:0/4152724483 >> 172.21.15.7:0/4152724483 conn(0x7fe7f4071830 msgr2=0x7fe7f40a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:13.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7fa433700 1 -- 172.21.15.7:0/4152724483 shutdown_connections 2023-09-30T01:56:13.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.591+0000 7fe7fa433700 1 -- 172.21.15.7:0/4152724483 wait complete. 2023-09-30T01:56:13.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.592+0000 7fe7fa433700 10 monclient: build_initial_monmap 2023-09-30T01:56:13.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.592+0000 7fe7fa433700 10 monclient: monmap: 2023-09-30T01:56:13.601 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:13.601 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:56:13.601 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:56:13.601 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:56:13.601 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:13.601 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:13.601 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:56:13.601 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:56:13.601 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:56:13.601 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:13.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.592+0000 7fe7fa433700 1 Processor -- start 2023-09-30T01:56:13.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.592+0000 7fe7fa433700 1 -- start start 2023-09-30T01:56:13.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.592+0000 7fe7fa433700 10 monclient: init 2023-09-30T01:56:13.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7fa433700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:13.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7fa433700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:56:13.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7fa433700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f4098540 0x7fe7f411ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:13.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7fa433700 10 monclient(hunting): picked mon.noname-a con 0x7fe7f4098540 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:56:13.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7fa433700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe7f4098910 0x7fe7f411bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:13.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7fa433700 10 monclient(hunting): picked mon.noname-c con 0x7fe7f4098910 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:13.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7fa433700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7f412ebc0 0x7fe7f411c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:13.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7fa433700 10 monclient(hunting): picked mon.noname-b con 0x7fe7f412ebc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:56:13.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7fa433700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:13.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7fa433700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f40a47e0 con 0x7fe7f4098540 2023-09-30T01:56:13.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7fa433700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:13.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7fa433700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f411f990 con 0x7fe7f412ebc0 2023-09-30T01:56:13.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7fa433700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:13.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7fa433700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f411eae0 con 0x7fe7f4098910 2023-09-30T01:56:13.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7fa433700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:13.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7fa433700 10 monclient(hunting): authenticate will time out at 1262.658467s 2023-09-30T01:56:13.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9431700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f4098540 0x7fe7f411ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:13.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9431700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f4098540 0x7fe7f411ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47406/0 (socket says 172.21.15.7:47406) 2023-09-30T01:56:13.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9431700 1 -- 172.21.15.7:0/3501124104 learned_addr learned my addr 172.21.15.7:0/3501124104 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:13.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9431700 10 monclient(hunting): get_auth_request con 0x7fe7f4098540 auth_method 0 2023-09-30T01:56:13.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9431700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:13.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9431700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:13.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9431700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:13.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f8c30700 1 --2- 172.21.15.7:0/3501124104 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe7f4098910 0x7fe7f411bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:13.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f8c30700 10 monclient(hunting): get_auth_request con 0x7fe7f4098910 auth_method 0 2023-09-30T01:56:13.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f8c30700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:13.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f8c30700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:13.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f8c30700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:13.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9431700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:13.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9431700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:13.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9431700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:13.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9c32700 1 --2- 172.21.15.7:0/3501124104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7f412ebc0 0x7fe7f411c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:13.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9c32700 10 monclient(hunting): get_auth_request con 0x7fe7f412ebc0 auth_method 0 2023-09-30T01:56:13.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9c32700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:13.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9c32700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:13.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9c32700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f8c30700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f8c30700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f8c30700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9c32700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9c32700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9c32700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9431700 10 monclient(hunting): handle_auth_done global_id 4422 payload 824 2023-09-30T01:56:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9431700 1 -- 172.21.15.7:0/3501124104 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe7f4098910 msgr2=0x7fe7f411bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9431700 1 --2- 172.21.15.7:0/3501124104 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe7f4098910 0x7fe7f411bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9431700 1 -- 172.21.15.7:0/3501124104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7f412ebc0 msgr2=0x7fe7f411c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9431700 1 --2- 172.21.15.7:0/3501124104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7f412ebc0 0x7fe7f411c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:13.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9431700 10 monclient: _finish_hunting 0 2023-09-30T01:56:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9431700 1 monclient: found mon.noname-a 2023-09-30T01:56:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9431700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9431700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:56:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.593+0000 7fe7f9431700 1 -- 172.21.15.7:0/3501124104 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e8009420 con 0x7fe7f4098540 2023-09-30T01:56:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7f9431700 1 -- 172.21.15.7:0/3501124104 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f4098540 msgr2=0x7fe7f411ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7f9431700 1 --2- 172.21.15.7:0/3501124104 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f4098540 0x7fe7f411ba20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4009fa0 tx=0x7fe7e4009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7f17fa700 1 -- 172.21.15.7:0/3501124104 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e400edd0 con 0x7fe7f4098540 2023-09-30T01:56:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7f17fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:56:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7f17fa700 10 monclient: dump: 2023-09-30T01:56:13.612 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:13.613 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:13.613 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7f17fa700 10 monclient: _finish_auth 0 2023-09-30T01:56:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7f17fa700 1 -- 172.21.15.7:0/3501124104 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e4014420 con 0x7fe7f4098540 2023-09-30T01:56:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7fa433700 5 monclient: authenticate success, global_id 4422 2023-09-30T01:56:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7fa433700 10 monclient: _renew_subs 2023-09-30T01:56:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7fa433700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7fa433700 1 -- 172.21.15.7:0/3501124104 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f40a05c0 con 0x7fe7f4098540 2023-09-30T01:56:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7f17fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:13.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7f17fa700 1 -- 172.21.15.7:0/3501124104 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e40147e0 con 0x7fe7f4098540 2023-09-30T01:56:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7f17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:56:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7f17fa700 10 monclient: dump: 2023-09-30T01:56:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:13.615 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:13.616 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7f37fe700 4 set_mon_vals no callback set 2023-09-30T01:56:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7fa433700 10 monclient: _renew_subs 2023-09-30T01:56:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7fa433700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.596+0000 7fe7fa433700 1 -- 172.21.15.7:0/3501124104 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f411dae0 con 0x7fe7f4098540 2023-09-30T01:56:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.597+0000 7fe7f17fa700 1 -- 172.21.15.7:0/3501124104 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fe7e4014b90 con 0x7fe7f4098540 2023-09-30T01:56:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.597+0000 7fe7fa433700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.597+0000 7fe7fa433700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.597+0000 7fe7fa433700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.597+0000 7fe7fa433700 1 -- 172.21.15.7:0/3501124104 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f406a4a0 con 0x7fe7f4098540 2023-09-30T01:56:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.597+0000 7fe7f17fa700 1 --2- 172.21.15.7:0/3501124104 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fe7e0035810 0x7fe7e0037cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:13.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.597+0000 7fe7f17fa700 1 -- 172.21.15.7:0/3501124104 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fe7e4048320 con 0x7fe7f4098540 2023-09-30T01:56:13.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.597+0000 7fe7f8c30700 1 --2- 172.21.15.7:0/3501124104 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fe7e0035810 0x7fe7e0037cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:13.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.597+0000 7fe7f8c30700 10 monclient: get_auth_request con 0x7fe7e0035810 auth_method 0 2023-09-30T01:56:13.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.600+0000 7fe7f8c30700 1 --2- 172.21.15.7:0/3501124104 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fe7e0035810 0x7fe7e0037cc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe7e80030c0 tx=0x7fe7e8016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:13.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.602+0000 7fe7f17fa700 1 -- 172.21.15.7:0/3501124104 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe7e4014e40 con 0x7fe7f4098540 2023-09-30T01:56:13.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.602+0000 7fe7f17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:13.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.602+0000 7fe7f17fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:56:13.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.836+0000 7fe7fa433700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:56:13.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.836+0000 7fe7fa433700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:13.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.836+0000 7fe7fa433700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:13.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.836+0000 7fe7fa433700 1 -- 172.21.15.7:0/3501124104 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe7f406a0a0 con 0x7fe7f4098540 2023-09-30T01:56:13.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.836+0000 7fe7f17fa700 1 -- 172.21.15.7:0/3501124104 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe7e401b310 con 0x7fe7f4098540 2023-09-30T01:56:13.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.836+0000 7fe7f17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:13.838 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.836+0000 7fe7f17fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:56:13.839 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:56:13.839 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:56:13.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.838+0000 7fe7fa433700 1 -- 172.21.15.7:0/3501124104 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fe7e0035810 msgr2=0x7fe7e0037cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:13.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.838+0000 7fe7fa433700 1 --2- 172.21.15.7:0/3501124104 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fe7e0035810 0x7fe7e0037cc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe7e80030c0 tx=0x7fe7e8016000 comp rx=0 tx=0).stop 2023-09-30T01:56:13.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.838+0000 7fe7fa433700 10 monclient: shutdown 2023-09-30T01:56:13.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.838+0000 7fe7fa433700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:13.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.838+0000 7fe7fa433700 1 -- 172.21.15.7:0/3501124104 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f4098540 msgr2=0x7fe7f411ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:13.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.838+0000 7fe7fa433700 1 --2- 172.21.15.7:0/3501124104 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f4098540 0x7fe7f411ba20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4009fa0 tx=0x7fe7e4009880 comp rx=0 tx=0).stop 2023-09-30T01:56:13.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.838+0000 7fe7fa433700 1 -- 172.21.15.7:0/3501124104 shutdown_connections 2023-09-30T01:56:13.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.838+0000 7fe7fa433700 1 --2- 172.21.15.7:0/3501124104 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fe7e0035810 0x7fe7e0037cc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:13.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.838+0000 7fe7fa433700 1 --2- 172.21.15.7:0/3501124104 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7f4098540 0x7fe7f411ba20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:13.840 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.838+0000 7fe7fa433700 1 --2- 172.21.15.7:0/3501124104 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe7f412ebc0 0x7fe7f411c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:13.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.838+0000 7fe7fa433700 1 --2- 172.21.15.7:0/3501124104 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fe7f4098910 0x7fe7f411bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:13.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.838+0000 7fe7fa433700 1 -- 172.21.15.7:0/3501124104 >> 172.21.15.7:0/3501124104 conn(0x7fe7f4071830 msgr2=0x7fe7f40a1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:13.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.838+0000 7fe7fa433700 1 -- 172.21.15.7:0/3501124104 shutdown_connections 2023-09-30T01:56:13.841 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:13.838+0000 7fe7fa433700 1 -- 172.21.15.7:0/3501124104 wait complete. 2023-09-30T01:56:16.851 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:56:16.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.982+0000 7fcc2eacc700 10 monclient: get_monmap_and_config 2023-09-30T01:56:16.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.982+0000 7fcc2eacc700 10 monclient: build_initial_monmap 2023-09-30T01:56:16.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.982+0000 7fcc2eacc700 10 monclient: monmap: 2023-09-30T01:56:16.982 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:16.982 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:56:16.983286+0000 2023-09-30T01:56:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:56:16.983286+0000 2023-09-30T01:56:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:56:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:56:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:56:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:56:16.983 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:56:16.983 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:16.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.982+0000 7fcc2eacc700 1 Processor -- start 2023-09-30T01:56:16.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2eacc700 1 -- start start 2023-09-30T01:56:16.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2eacc700 10 monclient: init 2023-09-30T01:56:16.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2eacc700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:16.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2eacc700 10 monclient: _add_conns ranks=[3,1,0,2,4] 2023-09-30T01:56:16.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2eacc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcc28096a50 0x7fcc28132680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:16.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2eacc700 10 monclient(hunting): picked mon.noname-a con 0x7fcc28096a50 addr v2:172.21.15.7:3300/0 2023-09-30T01:56:16.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2eacc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc28096e20 addr v1:172.21.15.50:6789/0 2023-09-30T01:56:16.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2eacc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcc28137710 addr v1:172.21.15.7:6789/0 2023-09-30T01:56:16.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2eacc700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc28099440 con 0x7fcc28137710 2023-09-30T01:56:16.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2eacc700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc28097b40 con 0x7fcc28096e20 2023-09-30T01:56:16.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2eacc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:16.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2eacc700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc28099160 con 0x7fcc28096a50 2023-09-30T01:56:16.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2eacc700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:16.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2eacc700 10 monclient(hunting): authenticate will time out at 1266.048608s 2023-09-30T01:56:16.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2c868700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fcc28096a50 0x7fcc28132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:16.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2d069700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fcc28137710 0x7fcc28139af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53472/0 (socket says 172.21.15.7:53472) 2023-09-30T01:56:16.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2d069700 1 -- 172.21.15.7:0/1124510591 learned_addr learned my addr 172.21.15.7:0/1124510591 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:56:16.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2c868700 10 monclient(hunting): get_auth_request con 0x7fcc28096a50 auth_method 0 2023-09-30T01:56:16.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2c868700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:16.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2c868700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:16.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2c868700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:16.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2c868700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:16.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2c868700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:16.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2c868700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:16.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2c868700 10 monclient(hunting): handle_auth_done global_id 4425 payload 274 2023-09-30T01:56:16.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2c868700 1 -- 172.21.15.7:0/1124510591 >> v1:172.21.15.50:6789/0 conn(0x7fcc28096e20 legacy=0x7fcc28097220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:16.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2c868700 1 -- 172.21.15.7:0/1124510591 >> v1:172.21.15.7:6789/0 conn(0x7fcc28137710 legacy=0x7fcc28139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:16.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2c868700 10 monclient: _finish_hunting 0 2023-09-30T01:56:16.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2c868700 1 monclient: found mon.noname-a 2023-09-30T01:56:16.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2c868700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:16.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2c868700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:56:16.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.983+0000 7fcc2c868700 1 -- 172.21.15.7:0/1124510591 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc28099720 con 0x7fcc28096a50 2023-09-30T01:56:16.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc2c868700 1 -- 172.21.15.7:0/1124510591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc28096a50 msgr2=0x7fcc28132680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:16.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc2c868700 1 --2- 172.21.15.7:0/1124510591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc28096a50 0x7fcc28132680 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcc28097d30 tx=0x7fcc180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22fbd25a796fc4d8 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:17.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc277fe700 1 -- 172.21.15.7:0/1124510591 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18011920 con 0x7fcc28096a50 2023-09-30T01:56:17.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:17.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc277fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:56:17.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc277fe700 10 monclient: dump: 2023-09-30T01:56:17.000 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:17.000 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:17.001 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:17.001 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:17.001 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:17.001 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:17.001 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:17.001 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:17.002 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:17.002 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:17.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc277fe700 10 monclient: _finish_auth 0 2023-09-30T01:56:17.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc277fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:17.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc2eacc700 5 monclient: authenticate success, global_id 4425 2023-09-30T01:56:17.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc2eacc700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:56:17.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc277fe700 1 -- 172.21.15.7:0/1124510591 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc18011a80 con 0x7fcc28096a50 2023-09-30T01:56:17.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc277fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:17.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc2eacc700 10 monclient: get_monmap_and_config success 2023-09-30T01:56:17.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc2eacc700 4 set_mon_vals no callback set 2023-09-30T01:56:17.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc277fe700 1 -- 172.21.15.7:0/1124510591 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18011e20 con 0x7fcc28096a50 2023-09-30T01:56:17.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:17.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc277fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:56:17.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc277fe700 10 monclient: dump: 2023-09-30T01:56:17.004 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:17.004 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:17.004 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:17.005 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:17.005 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:17.005 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:17.005 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:17.005 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:17.005 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:17.006 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:17.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc2eacc700 10 monclient: shutdown 2023-09-30T01:56:17.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc2eacc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:17.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc2eacc700 1 -- 172.21.15.7:0/1124510591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc28096a50 msgr2=0x7fcc28132680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:17.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc2eacc700 1 --2- 172.21.15.7:0/1124510591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc28096a50 0x7fcc28132680 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcc28097d30 tx=0x7fcc180091a0 comp rx=0 tx=0).stop 2023-09-30T01:56:17.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc2eacc700 1 -- 172.21.15.7:0/1124510591 shutdown_connections 2023-09-30T01:56:17.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc2eacc700 1 --2- 172.21.15.7:0/1124510591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc28096a50 0x7fcc28132680 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:17.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.986+0000 7fcc2eacc700 1 -- 172.21.15.7:0/1124510591 >> 172.21.15.7:0/1124510591 conn(0x7fcc28070cf0 msgr2=0x7fcc280710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:17.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.987+0000 7fcc2eacc700 1 -- 172.21.15.7:0/1124510591 shutdown_connections 2023-09-30T01:56:17.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.987+0000 7fcc2eacc700 1 -- 172.21.15.7:0/1124510591 wait complete. 2023-09-30T01:56:17.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.987+0000 7fcc2eacc700 10 monclient: build_initial_monmap 2023-09-30T01:56:17.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.987+0000 7fcc2eacc700 10 monclient: monmap: 2023-09-30T01:56:17.008 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:17.008 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:56:17.008 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:56:17.008 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:56:17.008 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:17.008 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:17.009 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:56:17.009 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:56:17.009 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:56:17.009 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:17.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.987+0000 7fcc2eacc700 1 Processor -- start 2023-09-30T01:56:17.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2eacc700 1 -- start start 2023-09-30T01:56:17.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2eacc700 10 monclient: init 2023-09-30T01:56:17.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2eacc700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:17.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2eacc700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:56:17.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2eacc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc28096a50 0x7fcc28141bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:17.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2eacc700 10 monclient(hunting): picked mon.noname-b con 0x7fcc28096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:56:17.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2eacc700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcc28096e20 0x7fcc2812f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:17.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2eacc700 10 monclient(hunting): picked mon.noname-c con 0x7fcc28096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:17.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2eacc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc28137710 0x7fcc2809df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:17.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2eacc700 10 monclient(hunting): picked mon.noname-a con 0x7fcc28137710 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:56:17.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2eacc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:17.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2eacc700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc281334e0 con 0x7fcc28137710 2023-09-30T01:56:17.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2eacc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:17.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2eacc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc28131de0 con 0x7fcc28096a50 2023-09-30T01:56:17.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2eacc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:17.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2eacc700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc280a16c0 con 0x7fcc28096e20 2023-09-30T01:56:17.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2eacc700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:17.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2eacc700 10 monclient(hunting): authenticate will time out at 1266.053697s 2023-09-30T01:56:17.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc27fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcc28096e20 0x7fcc2812f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:17.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc27fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcc28096e20 0x7fcc2812f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:56670/0 (socket says 172.21.15.7:56670) 2023-09-30T01:56:17.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2d069700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc28137710 0x7fcc2809df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:17.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc27fff700 1 -- 172.21.15.7:0/1912656966 learned_addr learned my addr 172.21.15.7:0/1912656966 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:17.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc27fff700 10 monclient(hunting): get_auth_request con 0x7fcc28096e20 auth_method 0 2023-09-30T01:56:17.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc27fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:17.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc27fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:17.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc27fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:17.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2d069700 10 monclient(hunting): get_auth_request con 0x7fcc28137710 auth_method 0 2023-09-30T01:56:17.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2d069700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:17.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2d069700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2d069700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2d069700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2d069700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2c868700 1 --2- 172.21.15.7:0/1912656966 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc28096a50 0x7fcc28141bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2d069700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc27fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:17.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc27fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:17.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc27fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:17.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2c868700 10 monclient(hunting): get_auth_request con 0x7fcc28096a50 auth_method 0 2023-09-30T01:56:17.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2c868700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:17.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2c868700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:17.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.988+0000 7fcc2c868700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:17.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.989+0000 7fcc2c868700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:17.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.989+0000 7fcc2c868700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:17.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.989+0000 7fcc2c868700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:17.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.989+0000 7fcc2d069700 10 monclient(hunting): handle_auth_done global_id 4428 payload 824 2023-09-30T01:56:17.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.989+0000 7fcc2d069700 1 -- 172.21.15.7:0/1912656966 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcc28096e20 msgr2=0x7fcc2812f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:17.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.989+0000 7fcc2d069700 1 --2- 172.21.15.7:0/1912656966 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcc28096e20 0x7fcc2812f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:17.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.989+0000 7fcc2d069700 1 -- 172.21.15.7:0/1912656966 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc28096a50 msgr2=0x7fcc28141bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:17.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.989+0000 7fcc2d069700 1 --2- 172.21.15.7:0/1912656966 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc28096a50 0x7fcc28141bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:17.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.989+0000 7fcc2d069700 10 monclient: _finish_hunting 0 2023-09-30T01:56:17.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.989+0000 7fcc2d069700 1 monclient: found mon.noname-a 2023-09-30T01:56:17.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.989+0000 7fcc2d069700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:17.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.989+0000 7fcc2d069700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:56:17.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.989+0000 7fcc2d069700 1 -- 172.21.15.7:0/1912656966 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc18009420 con 0x7fcc28137710 2023-09-30T01:56:17.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.989+0000 7fcc27fff700 1 --2- 172.21.15.7:0/1912656966 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcc28096e20 0x7fcc2812f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:56:17.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc2d069700 1 -- 172.21.15.7:0/1912656966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc28137710 msgr2=0x7fcc2809df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:17.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc2d069700 1 --2- 172.21.15.7:0/1912656966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc28137710 0x7fcc2809df60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcc2000cb60 tx=0x7fcc20005340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:17.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc24ff9700 1 -- 172.21.15.7:0/1912656966 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2000a9f0 con 0x7fcc28137710 2023-09-30T01:56:17.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc24ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:17.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc24ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:56:17.020 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc24ff9700 10 monclient: dump: 2023-09-30T01:56:17.020 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:17.020 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:17.020 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:17.020 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:17.020 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:17.020 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:17.020 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:17.021 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:17.021 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:17.021 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:17.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc24ff9700 10 monclient: _finish_auth 0 2023-09-30T01:56:17.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc24ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:17.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc2eacc700 5 monclient: authenticate success, global_id 4428 2023-09-30T01:56:17.021 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc24ff9700 1 -- 172.21.15.7:0/1912656966 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2000ab70 con 0x7fcc28137710 2023-09-30T01:56:17.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc24ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:17.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc24ff9700 1 -- 172.21.15.7:0/1912656966 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc200070f0 con 0x7fcc28137710 2023-09-30T01:56:17.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc2eacc700 10 monclient: _renew_subs 2023-09-30T01:56:17.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc2eacc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:17.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc2eacc700 1 -- 172.21.15.7:0/1912656966 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc280a19a0 con 0x7fcc28137710 2023-09-30T01:56:17.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc24ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:17.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc24ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:56:17.022 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc26ffd700 4 set_mon_vals no callback set 2023-09-30T01:56:17.023 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc24ff9700 10 monclient: dump: 2023-09-30T01:56:17.023 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:17.023 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:17.023 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:17.023 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:17.023 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:17.023 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:17.024 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:17.024 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:17.024 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:17.024 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:17.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc2eacc700 10 monclient: _renew_subs 2023-09-30T01:56:17.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc2eacc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:17.024 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.991+0000 7fcc2eacc700 1 -- 172.21.15.7:0/1912656966 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc28142130 con 0x7fcc28137710 2023-09-30T01:56:17.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.992+0000 7fcc24ff9700 1 -- 172.21.15.7:0/1912656966 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fcc200074e0 con 0x7fcc28137710 2023-09-30T01:56:17.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.992+0000 7fcc2eacc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:17.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.992+0000 7fcc2eacc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:17.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.992+0000 7fcc2eacc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:17.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.992+0000 7fcc2eacc700 1 -- 172.21.15.7:0/1912656966 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc280a1bc0 con 0x7fcc28137710 2023-09-30T01:56:17.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.992+0000 7fcc24ff9700 1 --2- 172.21.15.7:0/1912656966 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fcc140357e0 0x7fcc14037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:17.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.992+0000 7fcc24ff9700 1 -- 172.21.15.7:0/1912656966 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fcc20047fc0 con 0x7fcc28137710 2023-09-30T01:56:17.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.993+0000 7fcc2c868700 1 --2- 172.21.15.7:0/1912656966 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fcc140357e0 0x7fcc14037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:17.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.993+0000 7fcc2c868700 10 monclient: get_auth_request con 0x7fcc140357e0 auth_method 0 2023-09-30T01:56:17.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.995+0000 7fcc2c868700 1 --2- 172.21.15.7:0/1912656966 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fcc140357e0 0x7fcc14037c90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcc180091e0 tx=0x7fcc18016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:17.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.997+0000 7fcc24ff9700 1 -- 172.21.15.7:0/1912656966 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcc20007790 con 0x7fcc28137710 2023-09-30T01:56:17.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.997+0000 7fcc24ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:17.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:16.997+0000 7fcc24ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:56:17.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:17.232+0000 7fcc2eacc700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:56:17.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:17.232+0000 7fcc2eacc700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:17.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:17.232+0000 7fcc2eacc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:17.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:17.232+0000 7fcc2eacc700 1 -- 172.21.15.7:0/1912656966 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcc28064c10 con 0x7fcc28137710 2023-09-30T01:56:17.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:17.233+0000 7fcc24ff9700 1 -- 172.21.15.7:0/1912656966 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fcc20010300 con 0x7fcc28137710 2023-09-30T01:56:17.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:17.233+0000 7fcc24ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:17.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:17.233+0000 7fcc24ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:56:17.233 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:56:17.234 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:56:17.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:17.235+0000 7fcc2eacc700 1 -- 172.21.15.7:0/1912656966 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fcc140357e0 msgr2=0x7fcc14037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:17.235+0000 7fcc2eacc700 1 --2- 172.21.15.7:0/1912656966 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fcc140357e0 0x7fcc14037c90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcc180091e0 tx=0x7fcc18016000 comp rx=0 tx=0).stop 2023-09-30T01:56:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:17.235+0000 7fcc2eacc700 10 monclient: shutdown 2023-09-30T01:56:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:17.235+0000 7fcc2eacc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:17.235+0000 7fcc2eacc700 1 -- 172.21.15.7:0/1912656966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc28137710 msgr2=0x7fcc2809df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:17.235+0000 7fcc2eacc700 1 --2- 172.21.15.7:0/1912656966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc28137710 0x7fcc2809df60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcc2000cb60 tx=0x7fcc20005340 comp rx=0 tx=0).stop 2023-09-30T01:56:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:17.235+0000 7fcc2eacc700 1 -- 172.21.15.7:0/1912656966 shutdown_connections 2023-09-30T01:56:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:17.235+0000 7fcc2eacc700 1 --2- 172.21.15.7:0/1912656966 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fcc140357e0 0x7fcc14037c90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:17.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:17.235+0000 7fcc2eacc700 1 --2- 172.21.15.7:0/1912656966 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc28137710 0x7fcc2809df60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:17.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:17.235+0000 7fcc2eacc700 1 --2- 172.21.15.7:0/1912656966 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc28096a50 0x7fcc28141bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:17.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:17.235+0000 7fcc2eacc700 1 --2- 172.21.15.7:0/1912656966 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcc28096e20 0x7fcc2812f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:17.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:17.235+0000 7fcc2eacc700 1 -- 172.21.15.7:0/1912656966 >> 172.21.15.7:0/1912656966 conn(0x7fcc28070cf0 msgr2=0x7fcc280a4500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:17.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:17.235+0000 7fcc2eacc700 1 -- 172.21.15.7:0/1912656966 shutdown_connections 2023-09-30T01:56:17.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:17.235+0000 7fcc2eacc700 1 -- 172.21.15.7:0/1912656966 wait complete. 2023-09-30T01:56:20.248 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:56:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.380+0000 7f046f269700 10 monclient: get_monmap_and_config 2023-09-30T01:56:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.380+0000 7f046f269700 10 monclient: build_initial_monmap 2023-09-30T01:56:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.380+0000 7f046f269700 10 monclient: monmap: 2023-09-30T01:56:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:56:20.381556+0000 2023-09-30T01:56:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:56:20.381556+0000 2023-09-30T01:56:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:56:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:56:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:56:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:56:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:56:20.382 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:20.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046f269700 1 Processor -- start 2023-09-30T01:56:20.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046f269700 1 -- start start 2023-09-30T01:56:20.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046f269700 10 monclient: init 2023-09-30T01:56:20.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046f269700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:20.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046f269700 10 monclient: _add_conns ranks=[4,0,2,1,3] 2023-09-30T01:56:20.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046f269700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f04680a29e0 0x7f0468096a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046f269700 10 monclient(hunting): picked mon.noname-c con 0x7f04680a29e0 addr v2:172.21.15.50:3300/0 2023-09-30T01:56:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046f269700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0468096f90 addr v1:172.21.15.7:6789/0 2023-09-30T01:56:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046f269700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0468097390 0x7f0468132930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046f269700 10 monclient(hunting): picked mon.noname-b con 0x7f0468097390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046f269700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0468069cc0 con 0x7f0468096f90 2023-09-30T01:56:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046f269700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046f269700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0468098ea0 con 0x7f04680a29e0 2023-09-30T01:56:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046f269700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046f269700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04680a2db0 con 0x7f0468097390 2023-09-30T01:56:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046f269700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046f269700 10 monclient(hunting): authenticate will time out at 1269.446802s 2023-09-30T01:56:20.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046d806700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0468097390 0x7f0468132930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:20.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046d806700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0468097390 0x7f0468132930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:56680/0 (socket says 172.21.15.7:56680) 2023-09-30T01:56:20.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046d806700 1 -- 172.21.15.7:0/3820657539 learned_addr learned my addr 172.21.15.7:0/3820657539 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:20.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046d806700 10 monclient(hunting): get_auth_request con 0x7f0468097390 auth_method 0 2023-09-30T01:56:20.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046d806700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:20.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046d806700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:20.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.381+0000 7f046d806700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:20.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.382+0000 7f046d806700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:20.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.382+0000 7f046d806700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:20.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.382+0000 7f046d806700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:20.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.382+0000 7f046d005700 1 --2- 172.21.15.7:0/3820657539 >> v2:172.21.15.50:3300/0 conn(0x7f04680a29e0 0x7f0468096a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:20.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.382+0000 7f046d806700 10 monclient(hunting): handle_auth_done global_id 4439 payload 274 2023-09-30T01:56:20.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.382+0000 7f046d806700 1 -- 172.21.15.7:0/3820657539 >> v2:172.21.15.50:3300/0 conn(0x7f04680a29e0 msgr2=0x7f0468096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:20.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.382+0000 7f046d806700 1 --2- 172.21.15.7:0/3820657539 >> v2:172.21.15.50:3300/0 conn(0x7f04680a29e0 0x7f0468096a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:20.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.382+0000 7f046d806700 1 -- 172.21.15.7:0/3820657539 >> v1:172.21.15.7:6789/0 conn(0x7f0468096f90 legacy=0x7f0468132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:20.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.382+0000 7f046d806700 10 monclient: _finish_hunting 0 2023-09-30T01:56:20.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.382+0000 7f046d806700 1 monclient: found mon.noname-b 2023-09-30T01:56:20.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.382+0000 7f046d806700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:20.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.382+0000 7f046d806700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:56:20.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.382+0000 7f046d806700 1 -- 172.21.15.7:0/3820657539 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0468098270 con 0x7f0468097390 2023-09-30T01:56:20.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.382+0000 7f046d005700 10 monclient: get_auth_request con 0x7f04680a29e0 auth_method 0 2023-09-30T01:56:20.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.382+0000 7f046d005700 1 --2- 172.21.15.7:0/3820657539 >> v2:172.21.15.50:3300/0 conn(0x7f04680a29e0 0x7f0468096a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:56:20.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.384+0000 7f046d806700 1 -- 172.21.15.7:0/3820657539 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0468097390 msgr2=0x7f0468132930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:20.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.384+0000 7f046d806700 1 --2- 172.21.15.7:0/3820657539 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0468097390 0x7f0468132930 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0460009f20 tx=0x7f0460009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2208c820552ddaa server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:20.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.384+0000 7f045ffff700 1 -- 172.21.15.7:0/3820657539 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046000e070 con 0x7f0468097390 2023-09-30T01:56:20.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.384+0000 7f045ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:20.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.384+0000 7f045ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:56:20.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.384+0000 7f045ffff700 10 monclient: dump: 2023-09-30T01:56:20.395 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:20.395 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:20.395 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:20.395 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:20.395 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:20.395 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:20.395 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:20.395 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:20.395 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:20.395 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:20.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f045ffff700 10 monclient: _finish_auth 0 2023-09-30T01:56:20.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f045ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:20.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f046f269700 5 monclient: authenticate success, global_id 4439 2023-09-30T01:56:20.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f045ffff700 1 -- 172.21.15.7:0/3820657539 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046000bc50 con 0x7f0468097390 2023-09-30T01:56:20.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f046f269700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:56:20.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f045ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:20.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f046f269700 10 monclient: get_monmap_and_config success 2023-09-30T01:56:20.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f046f269700 4 set_mon_vals no callback set 2023-09-30T01:56:20.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f045ffff700 1 -- 172.21.15.7:0/3820657539 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0460012430 con 0x7f0468097390 2023-09-30T01:56:20.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f045ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:20.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f045ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:56:20.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f045ffff700 10 monclient: dump: 2023-09-30T01:56:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:20.397 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f046f269700 10 monclient: shutdown 2023-09-30T01:56:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f046f269700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f046f269700 1 -- 172.21.15.7:0/3820657539 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0468097390 msgr2=0x7f0468132930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f046f269700 1 --2- 172.21.15.7:0/3820657539 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0468097390 0x7f0468132930 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0460009f20 tx=0x7f0460009840 comp rx=0 tx=0).stop 2023-09-30T01:56:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f046f269700 1 -- 172.21.15.7:0/3820657539 shutdown_connections 2023-09-30T01:56:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f046f269700 1 --2- 172.21.15.7:0/3820657539 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0468097390 0x7f0468132930 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f046f269700 1 --2- 172.21.15.7:0/3820657539 >> v2:172.21.15.50:3300/0 conn(0x7f04680a29e0 0x7f0468096a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f046f269700 1 -- 172.21.15.7:0/3820657539 >> 172.21.15.7:0/3820657539 conn(0x7f04680710d0 msgr2=0x7f04680714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f046f269700 1 -- 172.21.15.7:0/3820657539 shutdown_connections 2023-09-30T01:56:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.385+0000 7f046f269700 1 -- 172.21.15.7:0/3820657539 wait complete. 2023-09-30T01:56:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.386+0000 7f046f269700 10 monclient: build_initial_monmap 2023-09-30T01:56:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.386+0000 7f046f269700 10 monclient: monmap: 2023-09-30T01:56:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:56:20.399 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:56:20.399 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:56:20.399 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:20.399 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:20.399 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:56:20.399 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:56:20.399 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:56:20.399 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:20.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.386+0000 7f046f269700 1 Processor -- start 2023-09-30T01:56:20.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.386+0000 7f046f269700 1 -- start start 2023-09-30T01:56:20.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.386+0000 7f046f269700 10 monclient: init 2023-09-30T01:56:20.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046f269700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:20.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046f269700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:56:20.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046f269700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0468096f90 0x7f04680a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:20.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046f269700 10 monclient(hunting): picked mon.noname-b con 0x7f0468096f90 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:56:20.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046f269700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0468097390 0x7f046809bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:20.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046f269700 10 monclient(hunting): picked mon.noname-a con 0x7f0468097390 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:56:20.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046f269700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f04680a29e0 0x7f046809c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:20.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046f269700 10 monclient(hunting): picked mon.noname-c con 0x7f04680a29e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:20.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046f269700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:20.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046f269700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04681350c0 con 0x7f0468097390 2023-09-30T01:56:20.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046f269700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:20.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046f269700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04680a0830 con 0x7f0468096f90 2023-09-30T01:56:20.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046f269700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:20.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046f269700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046809e3f0 con 0x7f04680a29e0 2023-09-30T01:56:20.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046f269700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:20.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046f269700 10 monclient(hunting): authenticate will time out at 1269.452175s 2023-09-30T01:56:20.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d806700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f04680a29e0 0x7f046809c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:20.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046c804700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0468097390 0x7f046809bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:20.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d806700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f04680a29e0 0x7f046809c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:56696/0 (socket says 172.21.15.7:56696) 2023-09-30T01:56:20.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046c804700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0468097390 0x7f046809bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42864/0 (socket says 172.21.15.7:42864) 2023-09-30T01:56:20.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d806700 1 -- 172.21.15.7:0/3191508085 learned_addr learned my addr 172.21.15.7:0/3191508085 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:20.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d806700 10 monclient(hunting): get_auth_request con 0x7f04680a29e0 auth_method 0 2023-09-30T01:56:20.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d806700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:20.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d806700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:20.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d806700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:20.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046c804700 10 monclient(hunting): get_auth_request con 0x7f0468097390 auth_method 0 2023-09-30T01:56:20.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046c804700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:20.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046c804700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:20.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046c804700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:20.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d005700 1 --2- 172.21.15.7:0/3191508085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0468096f90 0x7f04680a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:20.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d806700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:20.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d806700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:20.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d806700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:20.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046c804700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:20.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046c804700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:20.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046c804700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:20.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d005700 10 monclient(hunting): get_auth_request con 0x7f0468096f90 auth_method 0 2023-09-30T01:56:20.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d005700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:20.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d005700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:20.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d005700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:20.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d005700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:20.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d005700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:20.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d005700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d806700 10 monclient(hunting): handle_auth_done global_id 4442 payload 824 2023-09-30T01:56:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d806700 1 -- 172.21.15.7:0/3191508085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0468096f90 msgr2=0x7f04680a02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d806700 1 --2- 172.21.15.7:0/3191508085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0468096f90 0x7f04680a02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d806700 1 -- 172.21.15.7:0/3191508085 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0468097390 msgr2=0x7f046809bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d806700 1 --2- 172.21.15.7:0/3191508085 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0468097390 0x7f046809bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d806700 10 monclient: _finish_hunting 0 2023-09-30T01:56:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d806700 1 monclient: found mon.noname-c 2023-09-30T01:56:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d806700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d806700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:56:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046d806700 1 -- 172.21.15.7:0/3191508085 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0460009ae0 con 0x7f04680a29e0 2023-09-30T01:56:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.387+0000 7f046c804700 1 --2- 172.21.15.7:0/3191508085 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0468097390 0x7f046809bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:56:20.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.389+0000 7f046d806700 1 -- 172.21.15.7:0/3191508085 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f04680a29e0 msgr2=0x7f046809c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.389+0000 7f046d806700 1 --2- 172.21.15.7:0/3191508085 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f04680a29e0 0x7f046809c0c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f04680a2ef0 tx=0x7f0460017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.390+0000 7f045d7fa700 1 -- 172.21.15.7:0/3191508085 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0460012c60 con 0x7f04680a29e0 2023-09-30T01:56:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.390+0000 7f045d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.390+0000 7f045d7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:56:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.390+0000 7f045d7fa700 10 monclient: dump: 2023-09-30T01:56:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:20.405 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:20.406 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:20.406 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:20.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.390+0000 7f045d7fa700 10 monclient: _finish_auth 0 2023-09-30T01:56:20.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.390+0000 7f045d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:20.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.390+0000 7f045d7fa700 1 -- 172.21.15.7:0/3191508085 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0460012dc0 con 0x7f04680a29e0 2023-09-30T01:56:20.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.390+0000 7f046f269700 5 monclient: authenticate success, global_id 4442 2023-09-30T01:56:20.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.390+0000 7f045d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:20.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.390+0000 7f045d7fa700 1 -- 172.21.15.7:0/3191508085 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046001d670 con 0x7f04680a29e0 2023-09-30T01:56:20.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.390+0000 7f045d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:20.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.390+0000 7f045d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:56:20.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.390+0000 7f045d7fa700 10 monclient: dump: 2023-09-30T01:56:20.407 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:20.407 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:20.407 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:20.407 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:20.407 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:20.408 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:20.408 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:20.408 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:20.408 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:20.408 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:20.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.390+0000 7f046f269700 10 monclient: _renew_subs 2023-09-30T01:56:20.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.390+0000 7f046f269700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:20.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.390+0000 7f046f269700 1 -- 172.21.15.7:0/3191508085 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f046809f2e0 con 0x7f04680a29e0 2023-09-30T01:56:20.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.390+0000 7f045f7fe700 4 set_mon_vals no callback set 2023-09-30T01:56:20.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.390+0000 7f046f269700 10 monclient: _renew_subs 2023-09-30T01:56:20.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.390+0000 7f046f269700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:20.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.390+0000 7f046f269700 1 -- 172.21.15.7:0/3191508085 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f046809d7b0 con 0x7f04680a29e0 2023-09-30T01:56:20.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.391+0000 7f045d7fa700 1 -- 172.21.15.7:0/3191508085 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f046001d9e0 con 0x7f04680a29e0 2023-09-30T01:56:20.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.391+0000 7f046f269700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:20.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.391+0000 7f046f269700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:20.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.391+0000 7f046f269700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:20.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.391+0000 7f046f269700 1 -- 172.21.15.7:0/3191508085 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0468064c10 con 0x7f04680a29e0 2023-09-30T01:56:20.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.391+0000 7f045d7fa700 1 --2- 172.21.15.7:0/3191508085 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f044c035830 0x7f044c037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:20.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.391+0000 7f045d7fa700 1 -- 172.21.15.7:0/3191508085 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f04600493a0 con 0x7f04680a29e0 2023-09-30T01:56:20.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.391+0000 7f046d005700 1 --2- 172.21.15.7:0/3191508085 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f044c035830 0x7f044c037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:20.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.391+0000 7f046d005700 10 monclient: get_auth_request con 0x7f044c035830 auth_method 0 2023-09-30T01:56:20.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.394+0000 7f046d005700 1 --2- 172.21.15.7:0/3191508085 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f044c035830 0x7f044c037ce0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f046400ad40 tx=0x7f0464008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.396+0000 7f045d7fa700 1 -- 172.21.15.7:0/3191508085 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f04600076e0 con 0x7f04680a29e0 2023-09-30T01:56:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.396+0000 7f045d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:20.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.396+0000 7f045d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:56:20.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.631+0000 7f046f269700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:56:20.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.631+0000 7f046f269700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:20.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.631+0000 7f046f269700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:20.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.631+0000 7f046f269700 1 -- 172.21.15.7:0/3191508085 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0468099510 con 0x7f04680a29e0 2023-09-30T01:56:20.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.631+0000 7f045d7fa700 1 -- 172.21.15.7:0/3191508085 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0460009910 con 0x7f04680a29e0 2023-09-30T01:56:20.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.631+0000 7f045d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:20.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.631+0000 7f045d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:56:20.632 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:56:20.632 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:56:20.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.633+0000 7f046f269700 1 -- 172.21.15.7:0/3191508085 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f044c035830 msgr2=0x7f044c037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:20.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.633+0000 7f046f269700 1 --2- 172.21.15.7:0/3191508085 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f044c035830 0x7f044c037ce0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f046400ad40 tx=0x7f0464008040 comp rx=0 tx=0).stop 2023-09-30T01:56:20.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.633+0000 7f046f269700 10 monclient: shutdown 2023-09-30T01:56:20.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.633+0000 7f046f269700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:20.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.633+0000 7f046f269700 1 -- 172.21.15.7:0/3191508085 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f04680a29e0 msgr2=0x7f046809c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:20.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.633+0000 7f046f269700 1 --2- 172.21.15.7:0/3191508085 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f04680a29e0 0x7f046809c0c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f04680a2ef0 tx=0x7f0460017000 comp rx=0 tx=0).stop 2023-09-30T01:56:20.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.634+0000 7f046f269700 1 -- 172.21.15.7:0/3191508085 shutdown_connections 2023-09-30T01:56:20.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.634+0000 7f046f269700 1 --2- 172.21.15.7:0/3191508085 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f044c035830 0x7f044c037ce0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:20.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.634+0000 7f046f269700 1 --2- 172.21.15.7:0/3191508085 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f04680a29e0 0x7f046809c0c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:20.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.634+0000 7f046f269700 1 --2- 172.21.15.7:0/3191508085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0468096f90 0x7f04680a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:20.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.634+0000 7f046f269700 1 --2- 172.21.15.7:0/3191508085 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0468097390 0x7f046809bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:20.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.634+0000 7f046f269700 1 -- 172.21.15.7:0/3191508085 >> 172.21.15.7:0/3191508085 conn(0x7f04680710d0 msgr2=0x7f04680a4060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:20.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.634+0000 7f046f269700 1 -- 172.21.15.7:0/3191508085 shutdown_connections 2023-09-30T01:56:20.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:20.634+0000 7f046f269700 1 -- 172.21.15.7:0/3191508085 wait complete. 2023-09-30T01:56:23.647 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:56:23.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.776+0000 7f056a042700 10 monclient: get_monmap_and_config 2023-09-30T01:56:23.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.776+0000 7f056a042700 10 monclient: build_initial_monmap 2023-09-30T01:56:23.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.776+0000 7f056a042700 10 monclient: monmap: 2023-09-30T01:56:23.777 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:23.777 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:23.777 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:56:23.777873+0000 2023-09-30T01:56:23.777 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:56:23.777873+0000 2023-09-30T01:56:23.777 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:23.777 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:23.777 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:56:23.777 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:56:23.778 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:56:23.778 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:56:23.778 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:56:23.778 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:23.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.777+0000 7f056a042700 1 Processor -- start 2023-09-30T01:56:23.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.777+0000 7f056a042700 1 -- start start 2023-09-30T01:56:23.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.777+0000 7f056a042700 10 monclient: init 2023-09-30T01:56:23.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f056a042700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:23.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f056a042700 10 monclient: _add_conns ranks=[4,3,2,0,1] 2023-09-30T01:56:23.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f056a042700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0564096a50 0x7f05641327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:23.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f056a042700 10 monclient(hunting): picked mon.noname-c con 0x7f0564096a50 addr v2:172.21.15.50:3300/0 2023-09-30T01:56:23.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f056a042700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f05640a2390 0x7f0564096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:23.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f056a042700 10 monclient(hunting): picked mon.noname-a con 0x7f05640a2390 addr v2:172.21.15.7:3300/0 2023-09-30T01:56:23.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f056a042700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0564097360 0x7f056412ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:23.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f056a042700 10 monclient(hunting): picked mon.noname-b con 0x7f0564097360 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:23.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f056a042700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:23.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f056a042700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05640980e0 con 0x7f05640a2390 2023-09-30T01:56:23.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f056a042700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:23.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f056a042700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05640991a0 con 0x7f0564096a50 2023-09-30T01:56:23.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f056a042700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:23.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f056a042700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0564099480 con 0x7f0564097360 2023-09-30T01:56:23.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f056a042700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:23.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f056a042700 10 monclient(hunting): authenticate will time out at 1272.843140s 2023-09-30T01:56:23.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0563fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0564097360 0x7f056412ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:23.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0562ffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f05640a2390 0x7f0564096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:23.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0563fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0564097360 0x7f056412ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:47520/0 (socket says 172.21.15.7:47520) 2023-09-30T01:56:23.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0563fff700 1 -- 172.21.15.7:0/657435485 learned_addr learned my addr 172.21.15.7:0/657435485 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:23.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0563fff700 10 monclient(hunting): get_auth_request con 0x7f0564097360 auth_method 0 2023-09-30T01:56:23.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0563fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:23.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0563fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:23.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0563fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:23.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0562ffd700 10 monclient(hunting): get_auth_request con 0x7f05640a2390 auth_method 0 2023-09-30T01:56:23.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0562ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:23.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0562ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:23.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0562ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:23.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0563fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:23.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0563fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:23.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f05637fe700 1 --2- 172.21.15.7:0/657435485 >> v2:172.21.15.50:3300/0 conn(0x7f0564096a50 0x7f05641327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:23.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0563fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:23.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0562ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:23.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0562ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:23.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0562ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:23.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f05637fe700 10 monclient(hunting): get_auth_request con 0x7f0564096a50 auth_method 0 2023-09-30T01:56:23.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f05637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:23.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f05637fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:23.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f05637fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:23.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f05637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:23.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f05637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:23.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f05637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:23.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0562ffd700 10 monclient(hunting): handle_auth_done global_id 4434 payload 274 2023-09-30T01:56:23.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0562ffd700 1 -- 172.21.15.7:0/657435485 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0564097360 msgr2=0x7f056412ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:23.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0562ffd700 1 --2- 172.21.15.7:0/657435485 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0564097360 0x7f056412ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:23.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0562ffd700 1 -- 172.21.15.7:0/657435485 >> v2:172.21.15.50:3300/0 conn(0x7f0564096a50 msgr2=0x7f05641327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:23.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0562ffd700 1 --2- 172.21.15.7:0/657435485 >> v2:172.21.15.50:3300/0 conn(0x7f0564096a50 0x7f05641327a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:23.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0562ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:56:23.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0562ffd700 1 monclient: found mon.noname-a 2023-09-30T01:56:23.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0562ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:23.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0562ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:56:23.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0562ffd700 1 -- 172.21.15.7:0/657435485 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0564097bf0 con 0x7f05640a2390 2023-09-30T01:56:23.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.778+0000 7f0563fff700 1 --2- 172.21.15.7:0/657435485 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0564097360 0x7f056412ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:56:23.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.780+0000 7f0562ffd700 1 -- 172.21.15.7:0/657435485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05640a2390 msgr2=0x7f0564096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:23.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.780+0000 7f0562ffd700 1 --2- 172.21.15.7:0/657435485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05640a2390 0x7f0564096e20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f05640a2b00 tx=0x7f05580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4026f052f83cf7bf server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:23.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f05627fc700 1 -- 172.21.15.7:0/657435485 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0558011840 con 0x7f05640a2390 2023-09-30T01:56:23.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f05627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:23.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f05627fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:56:23.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f05627fc700 10 monclient: dump: 2023-09-30T01:56:23.788 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:23.788 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:23.788 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:23.788 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:23.789 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:23.789 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:23.789 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:23.789 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:23.789 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:23.789 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:23.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f05627fc700 10 monclient: _finish_auth 0 2023-09-30T01:56:23.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f05627fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:23.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f05627fc700 1 -- 172.21.15.7:0/657435485 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05580119c0 con 0x7f05640a2390 2023-09-30T01:56:23.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f056a042700 5 monclient: authenticate success, global_id 4434 2023-09-30T01:56:23.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f056a042700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:56:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f05627fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f056a042700 10 monclient: get_monmap_and_config success 2023-09-30T01:56:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f05627fc700 1 -- 172.21.15.7:0/657435485 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0558011d80 con 0x7f05640a2390 2023-09-30T01:56:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f05627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f056a042700 4 set_mon_vals no callback set 2023-09-30T01:56:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f05627fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:56:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f05627fc700 10 monclient: dump: 2023-09-30T01:56:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:23.790 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:23.791 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:23.791 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:23.791 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:23.791 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:23.791 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:23.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f056a042700 10 monclient: shutdown 2023-09-30T01:56:23.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f056a042700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:23.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f056a042700 1 -- 172.21.15.7:0/657435485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05640a2390 msgr2=0x7f0564096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:23.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f056a042700 1 --2- 172.21.15.7:0/657435485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05640a2390 0x7f0564096e20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f05640a2b00 tx=0x7f05580091a0 comp rx=0 tx=0).stop 2023-09-30T01:56:23.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f056a042700 1 -- 172.21.15.7:0/657435485 shutdown_connections 2023-09-30T01:56:23.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f056a042700 1 --2- 172.21.15.7:0/657435485 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0564097360 0x7f056412ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:23.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f056a042700 1 --2- 172.21.15.7:0/657435485 >> v2:172.21.15.50:3300/0 conn(0x7f0564096a50 0x7f05641327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:23.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f056a042700 1 --2- 172.21.15.7:0/657435485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f05640a2390 0x7f0564096e20 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:23.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f056a042700 1 -- 172.21.15.7:0/657435485 >> 172.21.15.7:0/657435485 conn(0x7f0564070cf0 msgr2=0x7f05640710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:23.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f056a042700 1 -- 172.21.15.7:0/657435485 shutdown_connections 2023-09-30T01:56:23.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.781+0000 7f056a042700 1 -- 172.21.15.7:0/657435485 wait complete. 2023-09-30T01:56:23.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.782+0000 7f056a042700 10 monclient: build_initial_monmap 2023-09-30T01:56:23.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.782+0000 7f056a042700 10 monclient: monmap: 2023-09-30T01:56:23.793 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:23.793 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:56:23.793 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:56:23.793 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:56:23.793 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:23.793 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:23.793 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:56:23.793 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:56:23.793 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:56:23.793 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:23.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.782+0000 7f056a042700 1 Processor -- start 2023-09-30T01:56:23.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.782+0000 7f056a042700 1 -- start start 2023-09-30T01:56:23.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.782+0000 7f056a042700 10 monclient: init 2023-09-30T01:56:23.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f056a042700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:23.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f056a042700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:56:23.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f056a042700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0564096a50 0x7f05640a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:23.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f056a042700 10 monclient(hunting): picked mon.noname-a con 0x7f0564096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:56:23.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f056a042700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0564097360 0x7f056411c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:23.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f056a042700 10 monclient(hunting): picked mon.noname-c con 0x7f0564097360 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:23.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f056a042700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05640a2390 0x7f056412e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:23.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f056a042700 10 monclient(hunting): picked mon.noname-b con 0x7f05640a2390 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:56:23.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f056a042700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:23.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f056a042700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05640a4d60 con 0x7f0564096a50 2023-09-30T01:56:23.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f056a042700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:23.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f056a042700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056411e5c0 con 0x7f05640a2390 2023-09-30T01:56:23.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f056a042700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:23.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f056a042700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f056411cfd0 con 0x7f0564097360 2023-09-30T01:56:23.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f056a042700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:23.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f056a042700 10 monclient(hunting): authenticate will time out at 1272.848284s 2023-09-30T01:56:23.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0562ffd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0564097360 0x7f056411c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:23.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f05637fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0564096a50 0x7f05640a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:23.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f05637fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0564096a50 0x7f05640a0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33020/0 (socket says 172.21.15.7:33020) 2023-09-30T01:56:23.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0562ffd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0564097360 0x7f056411c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:47524/0 (socket says 172.21.15.7:47524) 2023-09-30T01:56:23.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f05637fe700 1 -- 172.21.15.7:0/1380840035 learned_addr learned my addr 172.21.15.7:0/1380840035 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:23.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f05637fe700 10 monclient(hunting): get_auth_request con 0x7f0564096a50 auth_method 0 2023-09-30T01:56:23.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f05637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:23.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f05637fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:23.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f05637fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:23.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0562ffd700 10 monclient(hunting): get_auth_request con 0x7f0564097360 auth_method 0 2023-09-30T01:56:23.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0562ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:23.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0562ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:23.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0562ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:23.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0562ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:23.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0562ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:23.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0563fff700 1 --2- 172.21.15.7:0/1380840035 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05640a2390 0x7f056412e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:23.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0562ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:23.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f05637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:23.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f05637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:23.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f05637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:23.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0563fff700 10 monclient(hunting): get_auth_request con 0x7f05640a2390 auth_method 0 2023-09-30T01:56:23.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0563fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:23.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0563fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:23.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0563fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:23.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0563fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:23.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0563fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:23.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0563fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:23.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0562ffd700 10 monclient(hunting): handle_auth_done global_id 4448 payload 824 2023-09-30T01:56:23.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0562ffd700 1 -- 172.21.15.7:0/1380840035 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05640a2390 msgr2=0x7f056412e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:23.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0562ffd700 1 --2- 172.21.15.7:0/1380840035 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05640a2390 0x7f056412e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:23.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0562ffd700 1 -- 172.21.15.7:0/1380840035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0564096a50 msgr2=0x7f05640a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:23.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0562ffd700 1 --2- 172.21.15.7:0/1380840035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0564096a50 0x7f05640a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:23.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0562ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:56:23.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0562ffd700 1 monclient: found mon.noname-c 2023-09-30T01:56:23.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0562ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:23.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0562ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:56:23.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f0562ffd700 1 -- 172.21.15.7:0/1380840035 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055c009ae0 con 0x7f0564097360 2023-09-30T01:56:23.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.783+0000 7f05637fe700 1 --2- 172.21.15.7:0/1380840035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0564096a50 0x7f05640a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:56:23.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.785+0000 7f0562ffd700 1 -- 172.21.15.7:0/1380840035 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0564097360 msgr2=0x7f056411c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:23.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.785+0000 7f0562ffd700 1 --2- 172.21.15.7:0/1380840035 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0564097360 0x7f056411c2a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0558009fd0 tx=0x7f0558017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:23.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.786+0000 7f054b7fe700 1 -- 172.21.15.7:0/1380840035 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055801f070 con 0x7f0564097360 2023-09-30T01:56:23.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.786+0000 7f054b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:23.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.786+0000 7f054b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:56:23.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.786+0000 7f054b7fe700 10 monclient: dump: 2023-09-30T01:56:23.804 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:23.804 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:23.804 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:23.804 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:23.804 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:23.804 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:23.804 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:23.804 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:23.804 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:23.804 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:23.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.786+0000 7f054b7fe700 10 monclient: _finish_auth 0 2023-09-30T01:56:23.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.786+0000 7f054b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:23.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.786+0000 7f056a042700 5 monclient: authenticate success, global_id 4448 2023-09-30T01:56:23.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.786+0000 7f054b7fe700 1 -- 172.21.15.7:0/1380840035 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0558010820 con 0x7f0564097360 2023-09-30T01:56:23.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.786+0000 7f056a042700 10 monclient: _renew_subs 2023-09-30T01:56:23.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.786+0000 7f056a042700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:23.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.786+0000 7f056a042700 1 -- 172.21.15.7:0/1380840035 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0558009420 con 0x7f0564097360 2023-09-30T01:56:23.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.786+0000 7f054b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:23.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.786+0000 7f054b7fe700 1 -- 172.21.15.7:0/1380840035 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0558010c00 con 0x7f0564097360 2023-09-30T01:56:23.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.786+0000 7f054b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:23.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.786+0000 7f054b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:56:23.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.786+0000 7f054b7fe700 10 monclient: dump: 2023-09-30T01:56:23.806 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:23.806 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:23.806 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:23.806 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:23.806 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:23.806 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:23.806 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:23.806 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:23.806 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:23.806 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:23.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.786+0000 7f0561ffb700 4 set_mon_vals no callback set 2023-09-30T01:56:23.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.786+0000 7f056a042700 10 monclient: _renew_subs 2023-09-30T01:56:23.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.786+0000 7f056a042700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:23.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.786+0000 7f056a042700 1 -- 172.21.15.7:0/1380840035 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05640a1970 con 0x7f0564097360 2023-09-30T01:56:23.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.786+0000 7f054b7fe700 1 -- 172.21.15.7:0/1380840035 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f0558023070 con 0x7f0564097360 2023-09-30T01:56:23.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.787+0000 7f056a042700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:23.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.787+0000 7f056a042700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:23.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.787+0000 7f056a042700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:23.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.787+0000 7f056a042700 1 -- 172.21.15.7:0/1380840035 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0564064c10 con 0x7f0564097360 2023-09-30T01:56:23.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.787+0000 7f054b7fe700 1 --2- 172.21.15.7:0/1380840035 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f05500357e0 0x7f0550037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:23.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.787+0000 7f054b7fe700 1 -- 172.21.15.7:0/1380840035 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f0558048d30 con 0x7f0564097360 2023-09-30T01:56:23.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.787+0000 7f05637fe700 1 --2- 172.21.15.7:0/1380840035 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f05500357e0 0x7f0550037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:23.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.787+0000 7f05637fe700 10 monclient: get_auth_request con 0x7f05500357e0 auth_method 0 2023-09-30T01:56:23.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.789+0000 7f05637fe700 1 --2- 172.21.15.7:0/1380840035 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f05500357e0 0x7f0550037c90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0554009fd0 tx=0x7f05540091a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:23.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.791+0000 7f054b7fe700 1 -- 172.21.15.7:0/1380840035 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f055801a940 con 0x7f0564097360 2023-09-30T01:56:23.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.791+0000 7f054b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:23.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:23.791+0000 7f054b7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:56:24.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:24.030+0000 7f056a042700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:56:24.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:24.030+0000 7f056a042700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:24.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:24.030+0000 7f056a042700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:24.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:24.030+0000 7f056a042700 1 -- 172.21.15.7:0/1380840035 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0564132ce0 con 0x7f0564097360 2023-09-30T01:56:24.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:24.031+0000 7f054b7fe700 1 -- 172.21.15.7:0/1380840035 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f055802a080 con 0x7f0564097360 2023-09-30T01:56:24.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:24.031+0000 7f054b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:24.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:24.031+0000 7f054b7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:56:24.031 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:56:24.032 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:56:24.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:24.033+0000 7f056a042700 1 -- 172.21.15.7:0/1380840035 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f05500357e0 msgr2=0x7f0550037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:24.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:24.033+0000 7f056a042700 1 --2- 172.21.15.7:0/1380840035 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f05500357e0 0x7f0550037c90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0554009fd0 tx=0x7f05540091a0 comp rx=0 tx=0).stop 2023-09-30T01:56:24.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:24.033+0000 7f056a042700 10 monclient: shutdown 2023-09-30T01:56:24.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:24.033+0000 7f056a042700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:24.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:24.033+0000 7f056a042700 1 -- 172.21.15.7:0/1380840035 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0564097360 msgr2=0x7f056411c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:24.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:24.033+0000 7f056a042700 1 --2- 172.21.15.7:0/1380840035 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0564097360 0x7f056411c2a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0558009fd0 tx=0x7f0558017000 comp rx=0 tx=0).stop 2023-09-30T01:56:24.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:24.033+0000 7f056a042700 1 -- 172.21.15.7:0/1380840035 shutdown_connections 2023-09-30T01:56:24.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:24.033+0000 7f056a042700 1 --2- 172.21.15.7:0/1380840035 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f05500357e0 0x7f0550037c90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:24.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:24.033+0000 7f056a042700 1 --2- 172.21.15.7:0/1380840035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0564096a50 0x7f05640a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:24.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:24.033+0000 7f056a042700 1 --2- 172.21.15.7:0/1380840035 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05640a2390 0x7f056412e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:24.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:24.033+0000 7f056a042700 1 --2- 172.21.15.7:0/1380840035 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0564097360 0x7f056411c2a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:24.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:24.033+0000 7f056a042700 1 -- 172.21.15.7:0/1380840035 >> 172.21.15.7:0/1380840035 conn(0x7f0564070cf0 msgr2=0x7f0564130850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:24.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:24.033+0000 7f056a042700 1 -- 172.21.15.7:0/1380840035 shutdown_connections 2023-09-30T01:56:24.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:24.033+0000 7f056a042700 1 -- 172.21.15.7:0/1380840035 wait complete. 2023-09-30T01:56:27.047 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:56:27.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.178+0000 7f19038da700 10 monclient: get_monmap_and_config 2023-09-30T01:56:27.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.178+0000 7f19038da700 10 monclient: build_initial_monmap 2023-09-30T01:56:27.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.178+0000 7f19038da700 10 monclient: monmap: 2023-09-30T01:56:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:56:27.179539+0000 2023-09-30T01:56:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:56:27.179539+0000 2023-09-30T01:56:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:56:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:56:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:56:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:56:27.179 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:56:27.179 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:27.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.179+0000 7f19038da700 1 Processor -- start 2023-09-30T01:56:27.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.179+0000 7f19038da700 1 -- start start 2023-09-30T01:56:27.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.179+0000 7f19038da700 10 monclient: init 2023-09-30T01:56:27.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.179+0000 7f19038da700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:27.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.179+0000 7f19038da700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-09-30T01:56:27.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.179+0000 7f19038da700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f18fc096a50 0x7f18fc1327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:27.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.179+0000 7f19038da700 10 monclient(hunting): picked mon.noname-c con 0x7f18fc096a50 addr v2:172.21.15.50:3300/0 2023-09-30T01:56:27.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.179+0000 7f19038da700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f18fc0a2390 addr v1:172.21.15.50:6789/0 2023-09-30T01:56:27.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.179+0000 7f19038da700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18fc097530 0x7f18fc139720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:27.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.179+0000 7f19038da700 10 monclient(hunting): picked mon.noname-b con 0x7f18fc097530 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:27.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.179+0000 7f19038da700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18fc0980e0 con 0x7f18fc0a2390 2023-09-30T01:56:27.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.179+0000 7f19038da700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:27.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.179+0000 7f19038da700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18fc0991a0 con 0x7f18fc096a50 2023-09-30T01:56:27.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.179+0000 7f19038da700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:27.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.179+0000 7f19038da700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc099480 con 0x7f18fc097530 2023-09-30T01:56:27.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.179+0000 7f19038da700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:27.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.179+0000 7f19038da700 10 monclient(hunting): authenticate will time out at 1276.244792s 2023-09-30T01:56:27.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.179+0000 7f1901e77700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18fc097530 0x7f18fc139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:27.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.179+0000 7f1901e77700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18fc097530 0x7f18fc139720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:47530/0 (socket says 172.21.15.7:47530) 2023-09-30T01:56:27.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901e77700 1 -- 172.21.15.7:0/1743620492 learned_addr learned my addr 172.21.15.7:0/1743620492 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:27.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901e77700 10 monclient(hunting): get_auth_request con 0x7f18fc097530 auth_method 0 2023-09-30T01:56:27.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901e77700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:27.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901e77700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:27.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901e77700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:27.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901e77700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:27.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901676700 1 --2- 172.21.15.7:0/1743620492 >> v2:172.21.15.50:3300/0 conn(0x7f18fc096a50 0x7f18fc1327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:27.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901e77700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:27.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901e77700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:27.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901676700 10 monclient(hunting): get_auth_request con 0x7f18fc096a50 auth_method 0 2023-09-30T01:56:27.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901676700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:27.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901676700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:27.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901676700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:27.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901676700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:27.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901676700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:27.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901676700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:27.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901e77700 10 monclient(hunting): handle_auth_done global_id 4451 payload 274 2023-09-30T01:56:27.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901e77700 1 -- 172.21.15.7:0/1743620492 >> v2:172.21.15.50:3300/0 conn(0x7f18fc096a50 msgr2=0x7f18fc1327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:27.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901e77700 1 --2- 172.21.15.7:0/1743620492 >> v2:172.21.15.50:3300/0 conn(0x7f18fc096a50 0x7f18fc1327a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:27.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901e77700 1 -- 172.21.15.7:0/1743620492 >> v1:172.21.15.50:6789/0 conn(0x7f18fc0a2390 legacy=0x7f18fc096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:27.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901e77700 10 monclient: _finish_hunting 0 2023-09-30T01:56:27.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901e77700 1 monclient: found mon.noname-b 2023-09-30T01:56:27.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901e77700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:27.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901e77700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:56:27.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.180+0000 7f1901e77700 1 -- 172.21.15.7:0/1743620492 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18fc097bf0 con 0x7f18fc097530 2023-09-30T01:56:27.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.182+0000 7f1901e77700 1 -- 172.21.15.7:0/1743620492 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18fc097530 msgr2=0x7f18fc139720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:27.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.182+0000 7f1901e77700 1 --2- 172.21.15.7:0/1743620492 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18fc097530 0x7f18fc139720 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f18f40099b0 tx=0x7f18f4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b2fc4964d9e93 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:27.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.182+0000 7f18fbfff700 1 -- 172.21.15.7:0/1743620492 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f4009ea0 con 0x7f18fc097530 2023-09-30T01:56:27.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.182+0000 7f18fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:27.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.182+0000 7f18fbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:56:27.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.182+0000 7f18fbfff700 10 monclient: dump: 2023-09-30T01:56:27.195 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:27.195 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:27.195 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:27.196 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:27.196 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:27.196 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:27.196 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:27.196 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:27.196 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:27.196 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:27.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.182+0000 7f18fbfff700 10 monclient: _finish_auth 0 2023-09-30T01:56:27.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.182+0000 7f18fbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:27.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.182+0000 7f18fbfff700 1 -- 172.21.15.7:0/1743620492 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f400ba30 con 0x7f18fc097530 2023-09-30T01:56:27.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.182+0000 7f19038da700 5 monclient: authenticate success, global_id 4451 2023-09-30T01:56:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.182+0000 7f19038da700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:56:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.182+0000 7f18fbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.182+0000 7f18fbfff700 1 -- 172.21.15.7:0/1743620492 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f400bdf0 con 0x7f18fc097530 2023-09-30T01:56:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.182+0000 7f18fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.182+0000 7f18fbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:56:27.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.182+0000 7f18fbfff700 10 monclient: dump: 2023-09-30T01:56:27.198 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:27.198 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:27.198 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:27.198 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:27.198 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:27.198 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:27.201 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:27.201 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:27.201 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:27.202 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:27.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.182+0000 7f19038da700 10 monclient: get_monmap_and_config success 2023-09-30T01:56:27.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.182+0000 7f19038da700 4 set_mon_vals no callback set 2023-09-30T01:56:27.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.183+0000 7f19038da700 10 monclient: shutdown 2023-09-30T01:56:27.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.183+0000 7f19038da700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:27.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.183+0000 7f19038da700 1 -- 172.21.15.7:0/1743620492 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18fc097530 msgr2=0x7f18fc139720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:27.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.183+0000 7f19038da700 1 --2- 172.21.15.7:0/1743620492 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18fc097530 0x7f18fc139720 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f18f40099b0 tx=0x7f18f4009250 comp rx=0 tx=0).stop 2023-09-30T01:56:27.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.183+0000 7f19038da700 1 -- 172.21.15.7:0/1743620492 shutdown_connections 2023-09-30T01:56:27.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.183+0000 7f19038da700 1 --2- 172.21.15.7:0/1743620492 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18fc097530 0x7f18fc139720 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:27.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.183+0000 7f19038da700 1 --2- 172.21.15.7:0/1743620492 >> v2:172.21.15.50:3300/0 conn(0x7f18fc096a50 0x7f18fc1327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:27.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.183+0000 7f19038da700 1 -- 172.21.15.7:0/1743620492 >> 172.21.15.7:0/1743620492 conn(0x7f18fc070cf0 msgr2=0x7f18fc0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:27.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.183+0000 7f19038da700 1 -- 172.21.15.7:0/1743620492 shutdown_connections 2023-09-30T01:56:27.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.183+0000 7f19038da700 1 -- 172.21.15.7:0/1743620492 wait complete. 2023-09-30T01:56:27.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.183+0000 7f19038da700 10 monclient: build_initial_monmap 2023-09-30T01:56:27.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.183+0000 7f19038da700 10 monclient: monmap: 2023-09-30T01:56:27.203 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:27.203 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:56:27.203 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:56:27.203 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:56:27.203 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:27.203 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:27.203 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:56:27.203 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:56:27.203 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:56:27.204 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:27.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.184+0000 7f19038da700 1 Processor -- start 2023-09-30T01:56:27.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.184+0000 7f19038da700 1 -- start start 2023-09-30T01:56:27.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.184+0000 7f19038da700 10 monclient: init 2023-09-30T01:56:27.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.184+0000 7f19038da700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:27.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.184+0000 7f19038da700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:56:27.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.184+0000 7f19038da700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18fc096a50 0x7f18fc0a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:27.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.184+0000 7f19038da700 10 monclient(hunting): picked mon.noname-a con 0x7f18fc096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:56:27.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.184+0000 7f19038da700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18fc097530 0x7f18fc09b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:27.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.184+0000 7f19038da700 10 monclient(hunting): picked mon.noname-c con 0x7f18fc097530 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:27.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.184+0000 7f19038da700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18fc0a2390 0x7f18fc09b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:27.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.184+0000 7f19038da700 10 monclient(hunting): picked mon.noname-b con 0x7f18fc0a2390 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:56:27.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.184+0000 7f19038da700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:27.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.184+0000 7f19038da700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc1334e0 con 0x7f18fc096a50 2023-09-30T01:56:27.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.184+0000 7f19038da700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:27.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.184+0000 7f19038da700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc130350 con 0x7f18fc0a2390 2023-09-30T01:56:27.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.184+0000 7f19038da700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:27.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.184+0000 7f19038da700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc09f130 con 0x7f18fc097530 2023-09-30T01:56:27.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.184+0000 7f19038da700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:27.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.184+0000 7f19038da700 10 monclient(hunting): authenticate will time out at 1276.249965s 2023-09-30T01:56:27.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901676700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18fc096a50 0x7f18fc0a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:27.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1900e75700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18fc097530 0x7f18fc09b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:27.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901676700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18fc096a50 0x7f18fc0a1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33026/0 (socket says 172.21.15.7:33026) 2023-09-30T01:56:27.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901676700 1 -- 172.21.15.7:0/2383471547 learned_addr learned my addr 172.21.15.7:0/2383471547 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:27.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901676700 10 monclient(hunting): get_auth_request con 0x7f18fc096a50 auth_method 0 2023-09-30T01:56:27.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901676700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:27.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901676700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:27.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901676700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:27.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1900e75700 10 monclient(hunting): get_auth_request con 0x7f18fc097530 auth_method 0 2023-09-30T01:56:27.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1900e75700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:27.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1900e75700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:27.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1900e75700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:27.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901676700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:27.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901676700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:27.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901e77700 1 --2- 172.21.15.7:0/2383471547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18fc0a2390 0x7f18fc09b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:27.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901676700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:27.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1900e75700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:27.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1900e75700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:27.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1900e75700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:27.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901e77700 10 monclient(hunting): get_auth_request con 0x7f18fc0a2390 auth_method 0 2023-09-30T01:56:27.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901e77700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:27.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901e77700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:27.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901e77700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:27.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901676700 10 monclient(hunting): handle_auth_done global_id 4440 payload 824 2023-09-30T01:56:27.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901676700 1 -- 172.21.15.7:0/2383471547 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18fc097530 msgr2=0x7f18fc09b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:27.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901676700 1 --2- 172.21.15.7:0/2383471547 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18fc097530 0x7f18fc09b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:27.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901676700 1 -- 172.21.15.7:0/2383471547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18fc0a2390 msgr2=0x7f18fc09b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:27.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901676700 1 --2- 172.21.15.7:0/2383471547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18fc0a2390 0x7f18fc09b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:27.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901676700 10 monclient: _finish_hunting 0 2023-09-30T01:56:27.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901676700 1 monclient: found mon.noname-a 2023-09-30T01:56:27.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901676700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:27.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901676700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:56:27.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901676700 1 -- 172.21.15.7:0/2383471547 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f40094f0 con 0x7f18fc096a50 2023-09-30T01:56:27.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.185+0000 7f1901e77700 1 --2- 172.21.15.7:0/2383471547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18fc0a2390 0x7f18fc09b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:56:27.209 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f1901676700 1 -- 172.21.15.7:0/2383471547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18fc096a50 msgr2=0x7f18fc0a1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:27.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f1901676700 1 --2- 172.21.15.7:0/2383471547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18fc096a50 0x7f18fc0a1820 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f18ec009f10 tx=0x7f18ec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:27.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f18f97fa700 1 -- 172.21.15.7:0/2383471547 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec00cdd0 con 0x7f18fc096a50 2023-09-30T01:56:27.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f18f97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:27.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f18f97fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:56:27.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f18f97fa700 10 monclient: dump: 2023-09-30T01:56:27.210 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:27.210 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:27.210 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:27.211 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f18f97fa700 10 monclient: _finish_auth 0 2023-09-30T01:56:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f18f97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f18f97fa700 1 -- 172.21.15.7:0/2383471547 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18ec00f040 con 0x7f18fc096a50 2023-09-30T01:56:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f19038da700 5 monclient: authenticate success, global_id 4440 2023-09-30T01:56:27.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f19038da700 10 monclient: _renew_subs 2023-09-30T01:56:27.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f19038da700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:27.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f19038da700 1 -- 172.21.15.7:0/2383471547 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18fc09eb00 con 0x7f18fc096a50 2023-09-30T01:56:27.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f18f97fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:27.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f18f97fa700 1 -- 172.21.15.7:0/2383471547 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec0146d0 con 0x7f18fc096a50 2023-09-30T01:56:27.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f18f97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:27.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f18f97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:56:27.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f18f97fa700 10 monclient: dump: 2023-09-30T01:56:27.212 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:27.212 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:27.212 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:27.212 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:27.213 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f18fb7fe700 4 set_mon_vals no callback set 2023-09-30T01:56:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f19038da700 10 monclient: _renew_subs 2023-09-30T01:56:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f19038da700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.187+0000 7f19038da700 1 -- 172.21.15.7:0/2383471547 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18fc0a0e60 con 0x7f18fc096a50 2023-09-30T01:56:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.188+0000 7f18f97fa700 1 -- 172.21.15.7:0/2383471547 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f18ec014a80 con 0x7f18fc096a50 2023-09-30T01:56:27.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.188+0000 7f18f97fa700 1 --2- 172.21.15.7:0/2383471547 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f18e80357e0 0x7f18e8037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:27.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.188+0000 7f18f97fa700 1 -- 172.21.15.7:0/2383471547 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f18ec048770 con 0x7f18fc096a50 2023-09-30T01:56:27.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.188+0000 7f19038da700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:27.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.188+0000 7f19038da700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:27.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.188+0000 7f19038da700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:27.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.188+0000 7f19038da700 1 -- 172.21.15.7:0/2383471547 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18fc09d5b0 con 0x7f18fc096a50 2023-09-30T01:56:27.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.189+0000 7f1900e75700 1 --2- 172.21.15.7:0/2383471547 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f18e80357e0 0x7f18e8037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:27.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.189+0000 7f1900e75700 10 monclient: get_auth_request con 0x7f18e80357e0 auth_method 0 2023-09-30T01:56:27.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.191+0000 7f1900e75700 1 --2- 172.21.15.7:0/2383471547 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f18e80357e0 0x7f18e8037c90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f18f0003a30 tx=0x7f18f0007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:27.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.193+0000 7f18f97fa700 1 -- 172.21.15.7:0/2383471547 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f18fc09d5b0 con 0x7f18fc096a50 2023-09-30T01:56:27.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.193+0000 7f18f97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:27.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.193+0000 7f18f97fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:56:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.431+0000 7f19038da700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:56:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.431+0000 7f19038da700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.431+0000 7f19038da700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.431+0000 7f19038da700 1 -- 172.21.15.7:0/2383471547 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f18fc09d5b0 con 0x7f18fc096a50 2023-09-30T01:56:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.431+0000 7f18f97fa700 1 -- 172.21.15.7:0/2383471547 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f18ec02a100 con 0x7f18fc096a50 2023-09-30T01:56:27.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.431+0000 7f18f97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:27.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.431+0000 7f18f97fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:56:27.432 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:56:27.432 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:56:27.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.433+0000 7f19038da700 1 -- 172.21.15.7:0/2383471547 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f18e80357e0 msgr2=0x7f18e8037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:27.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.433+0000 7f19038da700 1 --2- 172.21.15.7:0/2383471547 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f18e80357e0 0x7f18e8037c90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f18f0003a30 tx=0x7f18f0007ea0 comp rx=0 tx=0).stop 2023-09-30T01:56:27.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.433+0000 7f19038da700 10 monclient: shutdown 2023-09-30T01:56:27.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.433+0000 7f19038da700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:27.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.433+0000 7f19038da700 1 -- 172.21.15.7:0/2383471547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18fc096a50 msgr2=0x7f18fc0a1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:27.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.433+0000 7f19038da700 1 --2- 172.21.15.7:0/2383471547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18fc096a50 0x7f18fc0a1820 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f18ec009f10 tx=0x7f18ec00b040 comp rx=0 tx=0).stop 2023-09-30T01:56:27.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.433+0000 7f19038da700 1 -- 172.21.15.7:0/2383471547 shutdown_connections 2023-09-30T01:56:27.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.433+0000 7f19038da700 1 --2- 172.21.15.7:0/2383471547 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f18e80357e0 0x7f18e8037c90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:27.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.433+0000 7f19038da700 1 --2- 172.21.15.7:0/2383471547 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f18fc096a50 0x7f18fc0a1820 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:27.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.433+0000 7f19038da700 1 --2- 172.21.15.7:0/2383471547 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f18fc0a2390 0x7f18fc09b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:27.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.433+0000 7f19038da700 1 --2- 172.21.15.7:0/2383471547 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f18fc097530 0x7f18fc09b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:27.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.433+0000 7f19038da700 1 -- 172.21.15.7:0/2383471547 >> 172.21.15.7:0/2383471547 conn(0x7f18fc070cf0 msgr2=0x7f18fc139100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:27.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.433+0000 7f19038da700 1 -- 172.21.15.7:0/2383471547 shutdown_connections 2023-09-30T01:56:27.434 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:27.433+0000 7f19038da700 1 -- 172.21.15.7:0/2383471547 wait complete. 2023-09-30T01:56:30.446 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:56:30.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.578+0000 7f6ef6a4b700 10 monclient: get_monmap_and_config 2023-09-30T01:56:30.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.578+0000 7f6ef6a4b700 10 monclient: build_initial_monmap 2023-09-30T01:56:30.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.578+0000 7f6ef6a4b700 10 monclient: monmap: 2023-09-30T01:56:30.578 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:30.578 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:30.579 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:56:30.579387+0000 2023-09-30T01:56:30.579 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:56:30.579387+0000 2023-09-30T01:56:30.579 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:30.579 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:30.579 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:56:30.579 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:56:30.579 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:56:30.579 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:56:30.579 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:56:30.580 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:30.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.579+0000 7f6ef6a4b700 1 Processor -- start 2023-09-30T01:56:30.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.579+0000 7f6ef6a4b700 1 -- start start 2023-09-30T01:56:30.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.579+0000 7f6ef6a4b700 10 monclient: init 2023-09-30T01:56:30.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.579+0000 7f6ef6a4b700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:30.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.579+0000 7f6ef6a4b700 10 monclient: _add_conns ranks=[4,0,1,3,2] 2023-09-30T01:56:30.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.579+0000 7f6ef6a4b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6ef00a29e0 0x7f6ef0096a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.579+0000 7f6ef6a4b700 10 monclient(hunting): picked mon.noname-c con 0x7f6ef00a29e0 addr v2:172.21.15.50:3300/0 2023-09-30T01:56:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.579+0000 7f6ef6a4b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ef0096f90 addr v1:172.21.15.7:6789/0 2023-09-30T01:56:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.579+0000 7f6ef6a4b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ef0097390 addr v1:172.21.15.50:6789/0 2023-09-30T01:56:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.579+0000 7f6ef6a4b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef0069cc0 con 0x7f6ef0096f90 2023-09-30T01:56:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.579+0000 7f6ef6a4b700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef00a2db0 con 0x7f6ef0097390 2023-09-30T01:56:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.579+0000 7f6ef6a4b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.579+0000 7f6ef6a4b700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ef0098ea0 con 0x7f6ef00a29e0 2023-09-30T01:56:30.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.579+0000 7f6ef6a4b700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:30.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.579+0000 7f6ef6a4b700 10 monclient(hunting): authenticate will time out at 1279.644949s 2023-09-30T01:56:30.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.580+0000 7f6eeffff700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6ef00a29e0 0x7f6ef0096a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:30.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.580+0000 7f6eef7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6ef0096f90 0x7f6ef0132220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60410/0 (socket says 172.21.15.7:60410) 2023-09-30T01:56:30.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.580+0000 7f6eef7fe700 1 -- 172.21.15.7:0/3372460909 learned_addr learned my addr 172.21.15.7:0/3372460909 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:56:30.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.580+0000 7f6eeffff700 10 monclient(hunting): get_auth_request con 0x7f6ef00a29e0 auth_method 0 2023-09-30T01:56:30.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.580+0000 7f6eeffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:30.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.580+0000 7f6eeffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:30.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.580+0000 7f6eeffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:30.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.580+0000 7f6eeffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:30.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.580+0000 7f6eeffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:30.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.580+0000 7f6eeffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:30.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.580+0000 7f6eeffff700 10 monclient(hunting): handle_auth_done global_id 4414 payload 274 2023-09-30T01:56:30.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.580+0000 7f6eeffff700 1 -- 172.21.15.7:0/3372460909 >> v1:172.21.15.50:6789/0 conn(0x7f6ef0097390 legacy=0x7f6ef0132930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:30.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.580+0000 7f6eeffff700 1 -- 172.21.15.7:0/3372460909 >> v1:172.21.15.7:6789/0 conn(0x7f6ef0096f90 legacy=0x7f6ef0132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:30.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.580+0000 7f6eeffff700 10 monclient: _finish_hunting 0 2023-09-30T01:56:30.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.580+0000 7f6eeffff700 1 monclient: found mon.noname-c 2023-09-30T01:56:30.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.580+0000 7f6eeffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:30.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.580+0000 7f6eeffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3300/0 2023-09-30T01:56:30.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.580+0000 7f6eeffff700 1 -- 172.21.15.7:0/3372460909 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef0098270 con 0x7f6ef00a29e0 2023-09-30T01:56:30.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.582+0000 7f6eeffff700 1 -- 172.21.15.7:0/3372460909 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ef00a29e0 msgr2=0x7f6ef0096a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:30.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.582+0000 7f6eeffff700 1 --2- 172.21.15.7:0/3372460909 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ef00a29e0 0x7f6ef0096a50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6ee00098d0 tx=0x7f6ee0009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ebe830c20220ac6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:30.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6eeeffd700 1 -- 172.21.15.7:0/3372460909 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee000b9f0 con 0x7f6ef00a29e0 2023-09-30T01:56:30.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6eeeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:30.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6eeeffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:56:30.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6eeeffd700 10 monclient: dump: 2023-09-30T01:56:30.595 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:30.595 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:30.595 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:30.595 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:30.595 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:30.595 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:30.595 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:30.596 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:30.596 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:30.596 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:30.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6eeeffd700 10 monclient: _finish_auth 0 2023-09-30T01:56:30.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6eeeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:30.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6eeeffd700 1 -- 172.21.15.7:0/3372460909 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee000bb50 con 0x7f6ef00a29e0 2023-09-30T01:56:30.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6ef6a4b700 5 monclient: authenticate success, global_id 4414 2023-09-30T01:56:30.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6ef6a4b700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:56:30.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6eeeffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:30.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6ef6a4b700 10 monclient: get_monmap_and_config success 2023-09-30T01:56:30.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6eeeffd700 1 -- 172.21.15.7:0/3372460909 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee0011420 con 0x7f6ef00a29e0 2023-09-30T01:56:30.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6ef6a4b700 4 set_mon_vals no callback set 2023-09-30T01:56:30.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6eeeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:30.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6eeeffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:56:30.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6eeeffd700 10 monclient: dump: 2023-09-30T01:56:30.597 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:30.597 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:30.598 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:30.598 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:30.598 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:30.599 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:30.599 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:30.599 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:30.599 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:30.599 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:30.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6ef6a4b700 10 monclient: shutdown 2023-09-30T01:56:30.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6ef6a4b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:30.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6ef6a4b700 1 -- 172.21.15.7:0/3372460909 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ef00a29e0 msgr2=0x7f6ef0096a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:30.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6ef6a4b700 1 --2- 172.21.15.7:0/3372460909 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ef00a29e0 0x7f6ef0096a50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6ee00098d0 tx=0x7f6ee0009120 comp rx=0 tx=0).stop 2023-09-30T01:56:30.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6ef6a4b700 1 -- 172.21.15.7:0/3372460909 shutdown_connections 2023-09-30T01:56:30.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6ef6a4b700 1 --2- 172.21.15.7:0/3372460909 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ef00a29e0 0x7f6ef0096a50 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:30.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6ef6a4b700 1 -- 172.21.15.7:0/3372460909 >> 172.21.15.7:0/3372460909 conn(0x7f6ef00710d0 msgr2=0x7f6ef00714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:30.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6ef6a4b700 1 -- 172.21.15.7:0/3372460909 shutdown_connections 2023-09-30T01:56:30.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.583+0000 7f6ef6a4b700 1 -- 172.21.15.7:0/3372460909 wait complete. 2023-09-30T01:56:30.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.584+0000 7f6ef6a4b700 10 monclient: build_initial_monmap 2023-09-30T01:56:30.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.584+0000 7f6ef6a4b700 10 monclient: monmap: 2023-09-30T01:56:30.601 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:30.601 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:56:30.602 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:56:30.602 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:56:30.602 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:30.602 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:30.602 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:56:30.602 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:56:30.602 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:56:30.603 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:30.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.584+0000 7f6ef6a4b700 1 Processor -- start 2023-09-30T01:56:30.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.584+0000 7f6ef6a4b700 1 -- start start 2023-09-30T01:56:30.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.584+0000 7f6ef6a4b700 10 monclient: init 2023-09-30T01:56:30.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6ef6a4b700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:30.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6ef6a4b700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:56:30.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6ef6a4b700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6ef0096f90 0x7f6ef00a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:30.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6ef6a4b700 10 monclient(hunting): picked mon.noname-c con 0x7f6ef0096f90 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:30.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6ef6a4b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ef0097390 0x7f6ef0130d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:30.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6ef6a4b700 10 monclient(hunting): picked mon.noname-a con 0x7f6ef0097390 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:56:30.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6ef6a4b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ef00a29e0 0x7f6ef011bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:30.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6ef6a4b700 10 monclient(hunting): picked mon.noname-b con 0x7f6ef00a29e0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:56:30.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6ef6a4b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:30.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6ef6a4b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef011f4f0 con 0x7f6ef0097390 2023-09-30T01:56:30.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6ef6a4b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:30.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6ef6a4b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef00a1220 con 0x7f6ef00a29e0 2023-09-30T01:56:30.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6ef6a4b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:30.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6ef6a4b700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef0142390 con 0x7f6ef0096f90 2023-09-30T01:56:30.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6ef6a4b700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:30.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6ef6a4b700 10 monclient(hunting): authenticate will time out at 1279.650296s 2023-09-30T01:56:30.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eef7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ef0097390 0x7f6ef0130d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:30.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eeffff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6ef0096f90 0x7f6ef00a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:30.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eef7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ef0097390 0x7f6ef0130d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33042/0 (socket says 172.21.15.7:33042) 2023-09-30T01:56:30.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eeffff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6ef0096f90 0x7f6ef00a09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:47556/0 (socket says 172.21.15.7:47556) 2023-09-30T01:56:30.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eef7fe700 1 -- 172.21.15.7:0/83950204 learned_addr learned my addr 172.21.15.7:0/83950204 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:30.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eef7fe700 10 monclient(hunting): get_auth_request con 0x7f6ef0097390 auth_method 0 2023-09-30T01:56:30.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eef7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:30.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eef7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:30.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eef7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:30.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eeffff700 10 monclient(hunting): get_auth_request con 0x7f6ef0096f90 auth_method 0 2023-09-30T01:56:30.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eeffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:30.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eeffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:30.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eeffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:30.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eef7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:30.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eef7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:30.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eef7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:30.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eeffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:30.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eeffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eeffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6ef4fe8700 1 --2- 172.21.15.7:0/83950204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ef00a29e0 0x7f6ef011bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6ef4fe8700 10 monclient(hunting): get_auth_request con 0x7f6ef00a29e0 auth_method 0 2023-09-30T01:56:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6ef4fe8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6ef4fe8700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:30.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6ef4fe8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:30.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eeffff700 10 monclient(hunting): handle_auth_done global_id 4457 payload 824 2023-09-30T01:56:30.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eeffff700 1 -- 172.21.15.7:0/83950204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ef00a29e0 msgr2=0x7f6ef011bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:30.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eeffff700 1 --2- 172.21.15.7:0/83950204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ef00a29e0 0x7f6ef011bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:30.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eeffff700 1 -- 172.21.15.7:0/83950204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ef0097390 msgr2=0x7f6ef0130d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:30.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eeffff700 1 --2- 172.21.15.7:0/83950204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ef0097390 0x7f6ef0130d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:30.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eeffff700 10 monclient: _finish_hunting 0 2023-09-30T01:56:30.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eeffff700 1 monclient: found mon.noname-c 2023-09-30T01:56:30.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eeffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:30.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eeffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:56:30.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eeffff700 1 -- 172.21.15.7:0/83950204 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee00093c0 con 0x7f6ef0096f90 2023-09-30T01:56:30.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.585+0000 7f6eef7fe700 1 --2- 172.21.15.7:0/83950204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ef0097390 0x7f6ef0130d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:56:30.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6eeffff700 1 -- 172.21.15.7:0/83950204 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6ef0096f90 msgr2=0x7f6ef00a09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:30.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6eeffff700 1 --2- 172.21.15.7:0/83950204 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6ef0096f90 0x7f6ef00a09d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0015000 tx=0x7f6ee0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:30.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6ed7fff700 1 -- 172.21.15.7:0/83950204 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee0006ad0 con 0x7f6ef0096f90 2023-09-30T01:56:30.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6ed7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:30.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6ed7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:56:30.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6ed7fff700 10 monclient: dump: 2023-09-30T01:56:30.613 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:30.613 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:30.613 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:30.613 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:30.613 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:30.614 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:30.614 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:30.614 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:30.614 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:30.614 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:30.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6ed7fff700 10 monclient: _finish_auth 0 2023-09-30T01:56:30.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6ed7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:30.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6ef6a4b700 5 monclient: authenticate success, global_id 4457 2023-09-30T01:56:30.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6ed7fff700 1 -- 172.21.15.7:0/83950204 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee000bed0 con 0x7f6ef0096f90 2023-09-30T01:56:30.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6ef6a4b700 10 monclient: _renew_subs 2023-09-30T01:56:30.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6ef6a4b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:30.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6ef6a4b700 1 -- 172.21.15.7:0/83950204 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ef011ee30 con 0x7f6ef0096f90 2023-09-30T01:56:30.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6ed7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:30.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6ed7fff700 1 -- 172.21.15.7:0/83950204 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee001b880 con 0x7f6ef0096f90 2023-09-30T01:56:30.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6ed7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:30.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6ed7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:56:30.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6ed7fff700 10 monclient: dump: 2023-09-30T01:56:30.616 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:30.617 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:30.617 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:30.617 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:30.617 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:30.617 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:30.617 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:30.617 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:30.618 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:30.618 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:30.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6eee7fc700 4 set_mon_vals no callback set 2023-09-30T01:56:30.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6ef6a4b700 10 monclient: _renew_subs 2023-09-30T01:56:30.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6ef6a4b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:30.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.588+0000 7f6ef6a4b700 1 -- 172.21.15.7:0/83950204 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ef011f1c0 con 0x7f6ef0096f90 2023-09-30T01:56:30.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.589+0000 7f6ed7fff700 1 -- 172.21.15.7:0/83950204 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f6ee001bcf0 con 0x7f6ef0096f90 2023-09-30T01:56:30.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.589+0000 7f6ef6a4b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:30.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.589+0000 7f6ef6a4b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:30.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.589+0000 7f6ef6a4b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:30.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.589+0000 7f6ef6a4b700 1 -- 172.21.15.7:0/83950204 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ef011d420 con 0x7f6ef0096f90 2023-09-30T01:56:30.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.589+0000 7f6ed7fff700 1 --2- 172.21.15.7:0/83950204 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6edc0357e0 0x7f6edc037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:30.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.589+0000 7f6ed7fff700 1 -- 172.21.15.7:0/83950204 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f6ee0049cb0 con 0x7f6ef0096f90 2023-09-30T01:56:30.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.589+0000 7f6eef7fe700 1 --2- 172.21.15.7:0/83950204 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6edc0357e0 0x7f6edc037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:30.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.589+0000 7f6eef7fe700 10 monclient: get_auth_request con 0x7f6edc0357e0 auth_method 0 2023-09-30T01:56:30.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.589+0000 7f6eef7fe700 1 -- 172.21.15.7:0/83950204 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6edc0357e0 msgr2=0x7f6edc037c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-09-30T01:56:30.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.589+0000 7f6eef7fe700 1 -- 172.21.15.7:0/83950204 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6edc0357e0 msgr2=0x7f6edc037c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-30T01:56:30.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.589+0000 7f6eef7fe700 1 --2- 172.21.15.7:0/83950204 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6edc0357e0 0x7f6edc037c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:56:30.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.589+0000 7f6eef7fe700 1 --2- 172.21.15.7:0/83950204 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6edc0357e0 0x7f6edc037c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:56:30.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.594+0000 7f6ed7fff700 1 -- 172.21.15.7:0/83950204 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6ee00237a0 con 0x7f6ef0096f90 2023-09-30T01:56:30.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.594+0000 7f6ed7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:30.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.594+0000 7f6ed7fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:56:30.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.790+0000 7f6eef7fe700 1 --2- 172.21.15.7:0/83950204 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6edc0357e0 0x7f6edc037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:30.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.790+0000 7f6eef7fe700 10 monclient: get_auth_request con 0x7f6edc0357e0 auth_method 0 2023-09-30T01:56:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.793+0000 7f6eef7fe700 1 --2- 172.21.15.7:0/83950204 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6edc0357e0 0x7f6edc037c90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6ef011cf20 tx=0x7f6ee4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:30.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.828+0000 7f6ef6a4b700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:56:30.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.828+0000 7f6ef6a4b700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:30.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.828+0000 7f6ef6a4b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:30.828 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.828+0000 7f6ef6a4b700 1 -- 172.21.15.7:0/83950204 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6ef011d420 con 0x7f6ef0096f90 2023-09-30T01:56:30.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.828+0000 7f6ed7fff700 1 -- 172.21.15.7:0/83950204 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6ee00243e0 con 0x7f6ef0096f90 2023-09-30T01:56:30.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.828+0000 7f6ed7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:30.829 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.828+0000 7f6ed7fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:56:30.829 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:56:30.829 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:56:30.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.830+0000 7f6ef6a4b700 1 -- 172.21.15.7:0/83950204 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6edc0357e0 msgr2=0x7f6edc037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:30.831 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.830+0000 7f6ef6a4b700 1 --2- 172.21.15.7:0/83950204 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6edc0357e0 0x7f6edc037c90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6ef011cf20 tx=0x7f6ee4009040 comp rx=0 tx=0).stop 2023-09-30T01:56:30.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.830+0000 7f6ef6a4b700 10 monclient: shutdown 2023-09-30T01:56:30.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.830+0000 7f6ef6a4b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:30.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.830+0000 7f6ef6a4b700 1 -- 172.21.15.7:0/83950204 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6ef0096f90 msgr2=0x7f6ef00a09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:30.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.830+0000 7f6ef6a4b700 1 --2- 172.21.15.7:0/83950204 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6ef0096f90 0x7f6ef00a09d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0015000 tx=0x7f6ee0016000 comp rx=0 tx=0).stop 2023-09-30T01:56:30.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.830+0000 7f6ef6a4b700 1 -- 172.21.15.7:0/83950204 shutdown_connections 2023-09-30T01:56:30.832 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.830+0000 7f6ef6a4b700 1 --2- 172.21.15.7:0/83950204 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6edc0357e0 0x7f6edc037c90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:30.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.830+0000 7f6ef6a4b700 1 --2- 172.21.15.7:0/83950204 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ef00a29e0 0x7f6ef011bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:30.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.830+0000 7f6ef6a4b700 1 --2- 172.21.15.7:0/83950204 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6ef0096f90 0x7f6ef00a09d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:30.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.830+0000 7f6ef6a4b700 1 --2- 172.21.15.7:0/83950204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ef0097390 0x7f6ef0130d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:30.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.830+0000 7f6ef6a4b700 1 -- 172.21.15.7:0/83950204 >> 172.21.15.7:0/83950204 conn(0x7f6ef00710d0 msgr2=0x7f6ef0132f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:30.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.830+0000 7f6ef6a4b700 1 -- 172.21.15.7:0/83950204 shutdown_connections 2023-09-30T01:56:30.833 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:30.830+0000 7f6ef6a4b700 1 -- 172.21.15.7:0/83950204 wait complete. 2023-09-30T01:56:33.843 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:56:33.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.974+0000 7f06a7a77700 10 monclient: get_monmap_and_config 2023-09-30T01:56:33.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.974+0000 7f06a7a77700 10 monclient: build_initial_monmap 2023-09-30T01:56:33.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.974+0000 7f06a7a77700 10 monclient: monmap: 2023-09-30T01:56:33.988 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:33.988 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:33.988 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:56:33.975086+0000 2023-09-30T01:56:33.989 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:56:33.975086+0000 2023-09-30T01:56:33.989 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:33.989 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:33.989 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:56:33.989 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:56:33.989 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:56:33.989 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:56:33.989 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:56:33.989 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:33.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.974+0000 7f06a7a77700 1 Processor -- start 2023-09-30T01:56:33.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.974+0000 7f06a7a77700 1 -- start start 2023-09-30T01:56:33.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.974+0000 7f06a7a77700 10 monclient: init 2023-09-30T01:56:33.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a7a77700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:33.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a7a77700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2023-09-30T01:56:33.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a7a77700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f06a0096a50 0x7f06a0132680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:33.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a7a77700 10 monclient(hunting): picked mon.noname-a con 0x7f06a0096a50 addr v2:172.21.15.7:3300/0 2023-09-30T01:56:33.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a7a77700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f06a0096e20 addr v1:172.21.15.7:6789/0 2023-09-30T01:56:33.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a7a77700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f06a0137710 0x7f06a0139af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:33.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a7a77700 10 monclient(hunting): picked mon.noname-b con 0x7f06a0137710 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:33.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a7a77700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a0099440 con 0x7f06a0096e20 2023-09-30T01:56:33.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a7a77700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:33.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a7a77700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06a0097b40 con 0x7f06a0096a50 2023-09-30T01:56:33.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a7a77700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:33.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a7a77700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a0099160 con 0x7f06a0137710 2023-09-30T01:56:33.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a7a77700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:33.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a7a77700 10 monclient(hunting): authenticate will time out at 1283.040325s 2023-09-30T01:56:33.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a6014700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f06a0137710 0x7f06a0139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:33.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a5813700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f06a0096a50 0x7f06a0132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:33.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a5012700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f06a0096e20 0x7f06a0097220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34474/0 (socket says 172.21.15.7:34474) 2023-09-30T01:56:33.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a5012700 1 -- 172.21.15.7:0/2796296388 learned_addr learned my addr 172.21.15.7:0/2796296388 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:56:33.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a6014700 10 monclient(hunting): get_auth_request con 0x7f06a0137710 auth_method 0 2023-09-30T01:56:33.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a6014700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:33.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a6014700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:33.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a6014700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:33.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a5813700 10 monclient(hunting): get_auth_request con 0x7f06a0096a50 auth_method 0 2023-09-30T01:56:33.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a5813700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:33.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a5813700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:33.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a5813700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:33.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a6014700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:33.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a6014700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:33.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a6014700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:33.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a5813700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:33.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a5813700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:33.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a5813700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:33.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a5813700 10 monclient(hunting): handle_auth_done global_id 4446 payload 274 2023-09-30T01:56:33.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a5813700 1 -- 172.21.15.7:0/2796296388 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f06a0137710 msgr2=0x7f06a0139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:33.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a5813700 1 --2- 172.21.15.7:0/2796296388 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f06a0137710 0x7f06a0139af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:33.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a5813700 1 -- 172.21.15.7:0/2796296388 >> v1:172.21.15.7:6789/0 conn(0x7f06a0096e20 legacy=0x7f06a0097220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:33.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a5813700 10 monclient: _finish_hunting 0 2023-09-30T01:56:33.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a5813700 1 monclient: found mon.noname-a 2023-09-30T01:56:33.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a5813700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:33.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a5813700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:56:33.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a5813700 1 -- 172.21.15.7:0/2796296388 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a0099720 con 0x7f06a0096a50 2023-09-30T01:56:33.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.975+0000 7f06a6014700 1 --2- 172.21.15.7:0/2796296388 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f06a0137710 0x7f06a0139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:56:33.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a5813700 1 -- 172.21.15.7:0/2796296388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06a0096a50 msgr2=0x7f06a0132680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:33.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a5813700 1 --2- 172.21.15.7:0/2796296388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06a0096a50 0x7f06a0132680 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f069c0098e0 tx=0x7f069c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae83c9f0e27172dd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:33.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a4811700 1 -- 172.21.15.7:0/2796296388 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069c011840 con 0x7f06a0096a50 2023-09-30T01:56:33.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a4811700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:33.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a4811700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:56:33.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a4811700 10 monclient: dump: 2023-09-30T01:56:33.997 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:33.997 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:33.997 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:33.997 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:33.997 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:33.997 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:33.998 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:33.998 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:33.998 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:33.998 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:33.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a4811700 10 monclient: _finish_auth 0 2023-09-30T01:56:33.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a4811700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:33.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a4811700 1 -- 172.21.15.7:0/2796296388 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069c0119a0 con 0x7f06a0096a50 2023-09-30T01:56:33.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a7a77700 5 monclient: authenticate success, global_id 4446 2023-09-30T01:56:33.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a7a77700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:56:33.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a4811700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:33.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a7a77700 10 monclient: get_monmap_and_config success 2023-09-30T01:56:33.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a4811700 1 -- 172.21.15.7:0/2796296388 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069c011d40 con 0x7f06a0096a50 2023-09-30T01:56:33.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a4811700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:33.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a7a77700 4 set_mon_vals no callback set 2023-09-30T01:56:33.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a4811700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:56:34.000 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a4811700 10 monclient: dump: 2023-09-30T01:56:34.000 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:34.000 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:34.000 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:34.000 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:34.000 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:34.001 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:34.001 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:34.001 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:34.001 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:34.001 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:34.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a7a77700 10 monclient: shutdown 2023-09-30T01:56:34.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a7a77700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:34.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a7a77700 1 -- 172.21.15.7:0/2796296388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06a0096a50 msgr2=0x7f06a0132680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:34.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a7a77700 1 --2- 172.21.15.7:0/2796296388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06a0096a50 0x7f06a0132680 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f069c0098e0 tx=0x7f069c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:56:34.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a7a77700 1 -- 172.21.15.7:0/2796296388 shutdown_connections 2023-09-30T01:56:34.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a7a77700 1 --2- 172.21.15.7:0/2796296388 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f06a0137710 0x7f06a0139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:34.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a7a77700 1 --2- 172.21.15.7:0/2796296388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06a0096a50 0x7f06a0132680 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:34.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a7a77700 1 -- 172.21.15.7:0/2796296388 >> 172.21.15.7:0/2796296388 conn(0x7f06a0070cf0 msgr2=0x7f06a00710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:34.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a7a77700 1 -- 172.21.15.7:0/2796296388 shutdown_connections 2023-09-30T01:56:34.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.978+0000 7f06a7a77700 1 -- 172.21.15.7:0/2796296388 wait complete. 2023-09-30T01:56:34.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.979+0000 7f06a7a77700 10 monclient: build_initial_monmap 2023-09-30T01:56:34.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.979+0000 7f06a7a77700 10 monclient: monmap: 2023-09-30T01:56:34.004 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:34.004 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:56:34.004 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:56:34.004 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:56:34.004 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:34.004 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:34.005 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:56:34.005 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:56:34.005 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:56:34.005 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:34.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.979+0000 7f06a7a77700 1 Processor -- start 2023-09-30T01:56:34.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.979+0000 7f06a7a77700 1 -- start start 2023-09-30T01:56:34.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.979+0000 7f06a7a77700 10 monclient: init 2023-09-30T01:56:34.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a7a77700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:34.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a7a77700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:56:34.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a7a77700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06a0096a50 0x7f06a00a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:34.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a7a77700 10 monclient(hunting): picked mon.noname-b con 0x7f06a0096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:56:34.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a7a77700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f06a0096e20 0x7f06a009cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:34.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a7a77700 10 monclient(hunting): picked mon.noname-c con 0x7f06a0096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:34.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a7a77700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06a0137710 0x7f06a009d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:34.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a7a77700 10 monclient(hunting): picked mon.noname-a con 0x7f06a0137710 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:56:34.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a7a77700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:34.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a7a77700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a009e690 con 0x7f06a0137710 2023-09-30T01:56:34.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a7a77700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:34.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a7a77700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a01334e0 con 0x7f06a0096a50 2023-09-30T01:56:34.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a7a77700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:34.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a7a77700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a0131db0 con 0x7f06a0096e20 2023-09-30T01:56:34.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a7a77700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:34.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a7a77700 10 monclient(hunting): authenticate will time out at 1283.045279s 2023-09-30T01:56:34.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5012700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f06a0096e20 0x7f06a009cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:34.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a6014700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06a0137710 0x7f06a009d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:34.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5012700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f06a0096e20 0x7f06a009cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:58438/0 (socket says 172.21.15.7:58438) 2023-09-30T01:56:34.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a6014700 10 monclient(hunting): get_auth_request con 0x7f06a0137710 auth_method 0 2023-09-30T01:56:34.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5012700 1 -- 172.21.15.7:0/1447222984 learned_addr learned my addr 172.21.15.7:0/1447222984 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:34.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a6014700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:34.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a6014700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:34.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a6014700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:34.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5012700 10 monclient(hunting): get_auth_request con 0x7f06a0096e20 auth_method 0 2023-09-30T01:56:34.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5012700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:34.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5012700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:34.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5012700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:34.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5813700 1 --2- 172.21.15.7:0/1447222984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06a0096a50 0x7f06a00a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:34.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5813700 10 monclient(hunting): get_auth_request con 0x7f06a0096a50 auth_method 0 2023-09-30T01:56:34.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5813700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:34.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5813700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:34.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5813700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:34.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5012700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:34.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5012700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:34.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5012700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:34.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a6014700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:34.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a6014700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:34.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a6014700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:34.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5813700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:34.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5813700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:34.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5813700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:34.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5012700 10 monclient(hunting): handle_auth_done global_id 4463 payload 824 2023-09-30T01:56:34.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5012700 1 -- 172.21.15.7:0/1447222984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06a0096a50 msgr2=0x7f06a00a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:34.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5012700 1 --2- 172.21.15.7:0/1447222984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06a0096a50 0x7f06a00a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:34.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5012700 1 -- 172.21.15.7:0/1447222984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06a0137710 msgr2=0x7f06a009d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:34.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5012700 1 --2- 172.21.15.7:0/1447222984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06a0137710 0x7f06a009d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:34.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5012700 10 monclient: _finish_hunting 0 2023-09-30T01:56:34.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5012700 1 monclient: found mon.noname-c 2023-09-30T01:56:34.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5012700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:34.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5012700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:56:34.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a5012700 1 -- 172.21.15.7:0/1447222984 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0698009a20 con 0x7f06a0096e20 2023-09-30T01:56:34.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.980+0000 7f06a6014700 1 --2- 172.21.15.7:0/1447222984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06a0137710 0x7f06a009d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:56:34.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.982+0000 7f06a5012700 1 -- 172.21.15.7:0/1447222984 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f06a0096e20 msgr2=0x7f06a009cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:34.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.982+0000 7f06a5012700 1 --2- 172.21.15.7:0/1447222984 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f06a0096e20 0x7f06a009cb00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f069000c5e0 tx=0x7f0690004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:34.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.983+0000 7f0695ffb700 1 -- 172.21.15.7:0/1447222984 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0690005d40 con 0x7f06a0096e20 2023-09-30T01:56:34.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.983+0000 7f0695ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:34.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.983+0000 7f0695ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:56:34.013 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.983+0000 7f0695ffb700 10 monclient: dump: 2023-09-30T01:56:34.013 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:34.013 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:34.013 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:34.013 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:34.014 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:34.014 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:34.014 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:34.014 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:34.014 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:34.014 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:34.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.983+0000 7f0695ffb700 10 monclient: _finish_auth 0 2023-09-30T01:56:34.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.983+0000 7f0695ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:34.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.983+0000 7f0695ffb700 1 -- 172.21.15.7:0/1447222984 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06900070d0 con 0x7f06a0096e20 2023-09-30T01:56:34.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.983+0000 7f06a7a77700 5 monclient: authenticate success, global_id 4463 2023-09-30T01:56:34.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.983+0000 7f06a7a77700 10 monclient: _renew_subs 2023-09-30T01:56:34.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.983+0000 7f06a7a77700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:34.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.983+0000 7f06a7a77700 1 -- 172.21.15.7:0/1447222984 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069c009420 con 0x7f06a0096e20 2023-09-30T01:56:34.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.983+0000 7f0695ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:34.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.983+0000 7f0695ffb700 1 -- 172.21.15.7:0/1447222984 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0690007490 con 0x7f06a0096e20 2023-09-30T01:56:34.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.983+0000 7f0695ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:34.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.983+0000 7f0695ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:56:34.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.983+0000 7f0695ffb700 10 monclient: dump: 2023-09-30T01:56:34.015 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:34.015 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:34.016 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:34.016 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:34.016 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:34.016 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:34.016 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:34.016 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:34.016 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:34.016 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:34.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.983+0000 7f0697fff700 4 set_mon_vals no callback set 2023-09-30T01:56:34.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.983+0000 7f06a7a77700 10 monclient: _renew_subs 2023-09-30T01:56:34.016 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.983+0000 7f06a7a77700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:34.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.983+0000 7f06a7a77700 1 -- 172.21.15.7:0/1447222984 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a009f2d0 con 0x7f06a0096e20 2023-09-30T01:56:34.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.984+0000 7f0695ffb700 1 -- 172.21.15.7:0/1447222984 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f069001b3b0 con 0x7f06a0096e20 2023-09-30T01:56:34.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.984+0000 7f06a7a77700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:34.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.984+0000 7f06a7a77700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:34.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.984+0000 7f06a7a77700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:34.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.984+0000 7f06a7a77700 1 -- 172.21.15.7:0/1447222984 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a0064c10 con 0x7f06a0096e20 2023-09-30T01:56:34.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.984+0000 7f0695ffb700 1 --2- 172.21.15.7:0/1447222984 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f06840357e0 0x7f0684037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:34.017 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.984+0000 7f0695ffb700 1 -- 172.21.15.7:0/1447222984 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f0690047260 con 0x7f06a0096e20 2023-09-30T01:56:34.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.984+0000 7f06a5813700 1 --2- 172.21.15.7:0/1447222984 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f06840357e0 0x7f0684037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:34.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.984+0000 7f06a5813700 10 monclient: get_auth_request con 0x7f06840357e0 auth_method 0 2023-09-30T01:56:34.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.987+0000 7f06a5813700 1 --2- 172.21.15.7:0/1447222984 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f06840357e0 0x7f0684037c90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f069c0030c0 tx=0x7f069c016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:34.018 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.989+0000 7f0695ffb700 1 -- 172.21.15.7:0/1447222984 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0690018260 con 0x7f06a0096e20 2023-09-30T01:56:34.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.989+0000 7f0695ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:34.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:33.989+0000 7f0695ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:56:34.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:34.230+0000 7f06a7a77700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:56:34.230 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:34.230+0000 7f06a7a77700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:34.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:34.230+0000 7f06a7a77700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:34.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:34.230+0000 7f06a7a77700 1 -- 172.21.15.7:0/1447222984 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f06a00a2440 con 0x7f06a0096e20 2023-09-30T01:56:34.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:34.230+0000 7f0695ffb700 1 -- 172.21.15.7:0/1447222984 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0690017100 con 0x7f06a0096e20 2023-09-30T01:56:34.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:34.230+0000 7f0695ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:34.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:34.230+0000 7f0695ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:56:34.232 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:56:34.232 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:56:34.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:34.232+0000 7f06a7a77700 1 -- 172.21.15.7:0/1447222984 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f06840357e0 msgr2=0x7f0684037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:34.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:34.232+0000 7f06a7a77700 1 --2- 172.21.15.7:0/1447222984 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f06840357e0 0x7f0684037c90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f069c0030c0 tx=0x7f069c016000 comp rx=0 tx=0).stop 2023-09-30T01:56:34.233 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:34.232+0000 7f06a7a77700 10 monclient: shutdown 2023-09-30T01:56:34.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:34.232+0000 7f06a7a77700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:34.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:34.232+0000 7f06a7a77700 1 -- 172.21.15.7:0/1447222984 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f06a0096e20 msgr2=0x7f06a009cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:34.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:34.232+0000 7f06a7a77700 1 --2- 172.21.15.7:0/1447222984 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f06a0096e20 0x7f06a009cb00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f069000c5e0 tx=0x7f0690004a80 comp rx=0 tx=0).stop 2023-09-30T01:56:34.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:34.232+0000 7f06a7a77700 1 -- 172.21.15.7:0/1447222984 shutdown_connections 2023-09-30T01:56:34.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:34.232+0000 7f06a7a77700 1 --2- 172.21.15.7:0/1447222984 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f06840357e0 0x7f0684037c90 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:34.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:34.232+0000 7f06a7a77700 1 --2- 172.21.15.7:0/1447222984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f06a0137710 0x7f06a009d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:34.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:34.232+0000 7f06a7a77700 1 --2- 172.21.15.7:0/1447222984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06a0096a50 0x7f06a00a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:34.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:34.232+0000 7f06a7a77700 1 --2- 172.21.15.7:0/1447222984 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f06a0096e20 0x7f06a009cb00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:34.234 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:34.233+0000 7f06a7a77700 1 -- 172.21.15.7:0/1447222984 >> 172.21.15.7:0/1447222984 conn(0x7f06a0070cf0 msgr2=0x7f06a012f110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:34.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:34.233+0000 7f06a7a77700 1 -- 172.21.15.7:0/1447222984 shutdown_connections 2023-09-30T01:56:34.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:34.233+0000 7f06a7a77700 1 -- 172.21.15.7:0/1447222984 wait complete. 2023-09-30T01:56:37.245 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:56:37.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.386+0000 7f2cd3eba700 10 monclient: get_monmap_and_config 2023-09-30T01:56:37.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.386+0000 7f2cd3eba700 10 monclient: build_initial_monmap 2023-09-30T01:56:37.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.386+0000 7f2cd3eba700 10 monclient: monmap: 2023-09-30T01:56:37.386 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:37.386 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:37.387 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:56:37.387354+0000 2023-09-30T01:56:37.387 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:56:37.387354+0000 2023-09-30T01:56:37.387 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:37.387 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:37.387 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:56:37.387 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:56:37.387 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:56:37.387 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:56:37.387 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:56:37.387 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:37.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.386+0000 7f2cd3eba700 1 Processor -- start 2023-09-30T01:56:37.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd3eba700 1 -- start start 2023-09-30T01:56:37.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd3eba700 10 monclient: init 2023-09-30T01:56:37.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd3eba700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:37.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd3eba700 10 monclient: _add_conns ranks=[3,1,0,4,2] 2023-09-30T01:56:37.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd3eba700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2ccc096a50 0x7f2ccc1327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:37.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd3eba700 10 monclient(hunting): picked mon.noname-a con 0x7f2ccc096a50 addr v2:172.21.15.7:3300/0 2023-09-30T01:56:37.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd3eba700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2ccc0a2390 addr v1:172.21.15.50:6789/0 2023-09-30T01:56:37.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd3eba700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ccc097530 addr v1:172.21.15.7:6789/0 2023-09-30T01:56:37.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd3eba700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ccc0980e0 con 0x7f2ccc097530 2023-09-30T01:56:37.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd3eba700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ccc0991a0 con 0x7f2ccc0a2390 2023-09-30T01:56:37.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd3eba700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:37.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd3eba700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ccc099480 con 0x7f2ccc096a50 2023-09-30T01:56:37.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd3eba700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:37.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd3eba700 10 monclient(hunting): authenticate will time out at 1286.452587s 2023-09-30T01:56:37.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd1c56700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2ccc096a50 0x7f2ccc1327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:37.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd2457700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2ccc097530 0x7f2ccc139720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34490/0 (socket says 172.21.15.7:34490) 2023-09-30T01:56:37.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd2457700 1 -- 172.21.15.7:0/2579896311 learned_addr learned my addr 172.21.15.7:0/2579896311 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:56:37.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd1c56700 10 monclient(hunting): get_auth_request con 0x7f2ccc096a50 auth_method 0 2023-09-30T01:56:37.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd1c56700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:37.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd1c56700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:37.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd1c56700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:37.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd1c56700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:37.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd1c56700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:37.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd1c56700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:37.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.387+0000 7f2cd1c56700 10 monclient(hunting): handle_auth_done global_id 4452 payload 274 2023-09-30T01:56:37.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.388+0000 7f2cd1c56700 1 -- 172.21.15.7:0/2579896311 >> v1:172.21.15.50:6789/0 conn(0x7f2ccc0a2390 legacy=0x7f2ccc096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:37.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.388+0000 7f2cd1c56700 1 -- 172.21.15.7:0/2579896311 >> v1:172.21.15.7:6789/0 conn(0x7f2ccc097530 legacy=0x7f2ccc139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:37.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.388+0000 7f2cd1c56700 10 monclient: _finish_hunting 0 2023-09-30T01:56:37.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.388+0000 7f2cd1c56700 1 monclient: found mon.noname-a 2023-09-30T01:56:37.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.388+0000 7f2cd1c56700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:37.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.388+0000 7f2cd1c56700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:56:37.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.388+0000 7f2cd1c56700 1 -- 172.21.15.7:0/2579896311 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ccc097bf0 con 0x7f2ccc096a50 2023-09-30T01:56:37.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd1c56700 1 -- 172.21.15.7:0/2579896311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ccc096a50 msgr2=0x7f2ccc1327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:37.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd1c56700 1 --2- 172.21.15.7:0/2579896311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ccc096a50 0x7f2ccc1327a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4009e60 tx=0x7f2cc4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c28d2e2a7b726fd6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:37.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd0c54700 1 -- 172.21.15.7:0/2579896311 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc400e070 con 0x7f2ccc096a50 2023-09-30T01:56:37.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd0c54700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:37.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd0c54700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:56:37.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd0c54700 10 monclient: dump: 2023-09-30T01:56:37.395 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:37.395 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:37.395 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:37.395 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:37.395 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:37.395 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:37.395 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:37.395 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:37.396 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:37.396 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:37.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd0c54700 10 monclient: _finish_auth 0 2023-09-30T01:56:37.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd0c54700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:37.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd0c54700 1 -- 172.21.15.7:0/2579896311 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc4012430 con 0x7f2ccc096a50 2023-09-30T01:56:37.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd3eba700 5 monclient: authenticate success, global_id 4452 2023-09-30T01:56:37.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd3eba700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:56:37.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd0c54700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:37.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd0c54700 1 -- 172.21.15.7:0/2579896311 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc400bea0 con 0x7f2ccc096a50 2023-09-30T01:56:37.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd3eba700 10 monclient: get_monmap_and_config success 2023-09-30T01:56:37.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd3eba700 4 set_mon_vals no callback set 2023-09-30T01:56:37.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd0c54700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:37.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd0c54700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:56:37.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd0c54700 10 monclient: dump: 2023-09-30T01:56:37.397 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:37.397 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:37.397 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:37.397 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:37.397 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:37.397 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:37.397 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:37.397 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:37.397 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:37.397 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:37.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd3eba700 10 monclient: shutdown 2023-09-30T01:56:37.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd3eba700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:37.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd3eba700 1 -- 172.21.15.7:0/2579896311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ccc096a50 msgr2=0x7f2ccc1327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:37.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.390+0000 7f2cd3eba700 1 --2- 172.21.15.7:0/2579896311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ccc096a50 0x7f2ccc1327a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4009e60 tx=0x7f2cc4009780 comp rx=0 tx=0).stop 2023-09-30T01:56:37.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.391+0000 7f2cd3eba700 1 -- 172.21.15.7:0/2579896311 shutdown_connections 2023-09-30T01:56:37.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.391+0000 7f2cd3eba700 1 --2- 172.21.15.7:0/2579896311 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ccc096a50 0x7f2ccc1327a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:37.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.391+0000 7f2cd3eba700 1 -- 172.21.15.7:0/2579896311 >> 172.21.15.7:0/2579896311 conn(0x7f2ccc070cf0 msgr2=0x7f2ccc0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:37.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.391+0000 7f2cd3eba700 1 -- 172.21.15.7:0/2579896311 shutdown_connections 2023-09-30T01:56:37.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.391+0000 7f2cd3eba700 1 -- 172.21.15.7:0/2579896311 wait complete. 2023-09-30T01:56:37.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.391+0000 7f2cd3eba700 10 monclient: build_initial_monmap 2023-09-30T01:56:37.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.391+0000 7f2cd3eba700 10 monclient: monmap: 2023-09-30T01:56:37.398 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:37.399 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:56:37.399 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:56:37.399 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:56:37.399 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:37.399 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:37.399 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:56:37.399 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:56:37.399 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:56:37.399 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:37.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.391+0000 7f2cd3eba700 1 Processor -- start 2023-09-30T01:56:37.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd3eba700 1 -- start start 2023-09-30T01:56:37.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd3eba700 10 monclient: init 2023-09-30T01:56:37.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd3eba700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:37.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd3eba700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:56:37.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd3eba700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2ccc096a50 0x7f2ccc09b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:37.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd3eba700 10 monclient(hunting): picked mon.noname-c con 0x7f2ccc096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:37.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd3eba700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ccc097530 0x7f2ccc09bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:37.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd3eba700 10 monclient(hunting): picked mon.noname-b con 0x7f2ccc097530 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:56:37.405 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd3eba700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ccc0a2390 0x7f2ccc09c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:37.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd3eba700 10 monclient(hunting): picked mon.noname-a con 0x7f2ccc0a2390 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:56:37.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd3eba700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:37.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd3eba700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc09eea0 con 0x7f2ccc0a2390 2023-09-30T01:56:37.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd3eba700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:37.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd3eba700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc09f180 con 0x7f2ccc097530 2023-09-30T01:56:37.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd3eba700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:37.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd3eba700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc09f460 con 0x7f2ccc096a50 2023-09-30T01:56:37.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd3eba700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:37.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd3eba700 10 monclient(hunting): authenticate will time out at 1286.457521s 2023-09-30T01:56:37.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd2457700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ccc0a2390 0x7f2ccc09c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:37.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd1c56700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2ccc096a50 0x7f2ccc09b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:37.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd2457700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ccc0a2390 0x7f2ccc09c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39890/0 (socket says 172.21.15.7:39890) 2023-09-30T01:56:37.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd1c56700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2ccc096a50 0x7f2ccc09b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:58454/0 (socket says 172.21.15.7:58454) 2023-09-30T01:56:37.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd2457700 1 -- 172.21.15.7:0/77559987 learned_addr learned my addr 172.21.15.7:0/77559987 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:37.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd2457700 10 monclient(hunting): get_auth_request con 0x7f2ccc0a2390 auth_method 0 2023-09-30T01:56:37.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd2457700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:37.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd2457700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:37.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd2457700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:37.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd1c56700 10 monclient(hunting): get_auth_request con 0x7f2ccc096a50 auth_method 0 2023-09-30T01:56:37.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd1c56700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:37.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd1c56700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:37.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd1c56700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:37.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd1455700 1 --2- 172.21.15.7:0/77559987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ccc097530 0x7f2ccc09bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:37.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd2457700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:37.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd2457700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:37.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd2457700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:37.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd1455700 10 monclient(hunting): get_auth_request con 0x7f2ccc097530 auth_method 0 2023-09-30T01:56:37.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd1455700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:37.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd1455700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:37.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd1455700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:37.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd1c56700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:37.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd1c56700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:37.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd1c56700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:37.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd1455700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:37.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd1455700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:37.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd1455700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:37.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd2457700 10 monclient(hunting): handle_auth_done global_id 4455 payload 824 2023-09-30T01:56:37.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd2457700 1 -- 172.21.15.7:0/77559987 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2ccc096a50 msgr2=0x7f2ccc09b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:37.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd2457700 1 --2- 172.21.15.7:0/77559987 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2ccc096a50 0x7f2ccc09b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:37.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd2457700 1 -- 172.21.15.7:0/77559987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ccc097530 msgr2=0x7f2ccc09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:37.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd2457700 1 --2- 172.21.15.7:0/77559987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ccc097530 0x7f2ccc09bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:37.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd2457700 10 monclient: _finish_hunting 0 2023-09-30T01:56:37.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd2457700 1 monclient: found mon.noname-a 2023-09-30T01:56:37.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd2457700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:37.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd2457700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:56:37.409 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd2457700 1 -- 172.21.15.7:0/77559987 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc4009a20 con 0x7f2ccc0a2390 2023-09-30T01:56:37.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.392+0000 7f2cd1c56700 1 --2- 172.21.15.7:0/77559987 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2ccc096a50 0x7f2ccc09b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:56:37.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2cd2457700 1 -- 172.21.15.7:0/77559987 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ccc0a2390 msgr2=0x7f2ccc09c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:37.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2cd2457700 1 --2- 172.21.15.7:0/77559987 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ccc0a2390 0x7f2ccc09c410 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc00c560 tx=0x7f2cbc004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:37.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2cc9ffb700 1 -- 172.21.15.7:0/77559987 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc004ab0 con 0x7f2ccc0a2390 2023-09-30T01:56:37.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2cc9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:37.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2cc9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:56:37.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2cc9ffb700 10 monclient: dump: 2023-09-30T01:56:37.410 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:37.410 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:37.411 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:37.411 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:37.411 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:37.411 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:37.411 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:37.411 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:37.411 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:37.411 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:37.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2cc9ffb700 10 monclient: _finish_auth 0 2023-09-30T01:56:37.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2cc9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:37.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2cd3eba700 5 monclient: authenticate success, global_id 4455 2023-09-30T01:56:37.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2cc9ffb700 1 -- 172.21.15.7:0/77559987 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cbc005d40 con 0x7f2ccc0a2390 2023-09-30T01:56:37.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2cc9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:37.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2cc9ffb700 1 -- 172.21.15.7:0/77559987 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc007320 con 0x7f2ccc0a2390 2023-09-30T01:56:37.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2cc9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:37.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2cc9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:56:37.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2cc9ffb700 10 monclient: dump: 2023-09-30T01:56:37.412 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:37.412 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:37.412 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:37.412 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:37.412 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:37.413 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:37.413 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:37.413 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:37.413 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:37.413 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:37.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2cd3eba700 10 monclient: _renew_subs 2023-09-30T01:56:37.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2cd3eba700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:37.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2cd3eba700 1 -- 172.21.15.7:0/77559987 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ccc09d370 con 0x7f2ccc0a2390 2023-09-30T01:56:37.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2ccbfff700 4 set_mon_vals no callback set 2023-09-30T01:56:37.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2cd3eba700 10 monclient: _renew_subs 2023-09-30T01:56:37.413 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2cd3eba700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:37.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.395+0000 7f2cd3eba700 1 -- 172.21.15.7:0/77559987 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ccc0a00c0 con 0x7f2ccc0a2390 2023-09-30T01:56:37.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.396+0000 7f2cc9ffb700 1 -- 172.21.15.7:0/77559987 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f2cbc01b3b0 con 0x7f2ccc0a2390 2023-09-30T01:56:37.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.396+0000 7f2cc9ffb700 1 --2- 172.21.15.7:0/77559987 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2cb80357e0 0x7f2cb8037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:37.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.396+0000 7f2cc9ffb700 1 -- 172.21.15.7:0/77559987 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f2cbc0470e0 con 0x7f2ccc0a2390 2023-09-30T01:56:37.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.396+0000 7f2cd3eba700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:37.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.396+0000 7f2cd3eba700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:37.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.396+0000 7f2cd3eba700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:37.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.396+0000 7f2cd3eba700 1 -- 172.21.15.7:0/77559987 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ccc097080 con 0x7f2ccc0a2390 2023-09-30T01:56:37.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.396+0000 7f2cd1c56700 1 --2- 172.21.15.7:0/77559987 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2cb80357e0 0x7f2cb8037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:37.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.396+0000 7f2cd1c56700 10 monclient: get_auth_request con 0x7f2cb80357e0 auth_method 0 2023-09-30T01:56:37.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.399+0000 7f2cd1c56700 1 --2- 172.21.15.7:0/77559987 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2cb80357e0 0x7f2cb8037c90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2cc40099a0 tx=0x7f2cc4018000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:37.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.401+0000 7f2cc9ffb700 1 -- 172.21.15.7:0/77559987 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2cbc017080 con 0x7f2ccc0a2390 2023-09-30T01:56:37.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.401+0000 7f2cc9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:37.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.401+0000 7f2cc9ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:56:37.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.634+0000 7f2cd3eba700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:56:37.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.634+0000 7f2cd3eba700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:37.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.634+0000 7f2cd3eba700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:37.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.634+0000 7f2cd3eba700 1 -- 172.21.15.7:0/77559987 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2ccc09dfb0 con 0x7f2ccc0a2390 2023-09-30T01:56:37.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.635+0000 7f2cc9ffb700 1 -- 172.21.15.7:0/77559987 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2cbc018e60 con 0x7f2ccc0a2390 2023-09-30T01:56:37.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.635+0000 7f2cc9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:37.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.635+0000 7f2cc9ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:56:37.635 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:56:37.636 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:56:37.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.637+0000 7f2cd3eba700 1 -- 172.21.15.7:0/77559987 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2cb80357e0 msgr2=0x7f2cb8037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:37.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.637+0000 7f2cd3eba700 1 --2- 172.21.15.7:0/77559987 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2cb80357e0 0x7f2cb8037c90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2cc40099a0 tx=0x7f2cc4018000 comp rx=0 tx=0).stop 2023-09-30T01:56:37.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.637+0000 7f2cd3eba700 10 monclient: shutdown 2023-09-30T01:56:37.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.637+0000 7f2cd3eba700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:37.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.637+0000 7f2cd3eba700 1 -- 172.21.15.7:0/77559987 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ccc0a2390 msgr2=0x7f2ccc09c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:37.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.637+0000 7f2cd3eba700 1 --2- 172.21.15.7:0/77559987 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ccc0a2390 0x7f2ccc09c410 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc00c560 tx=0x7f2cbc004a20 comp rx=0 tx=0).stop 2023-09-30T01:56:37.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.637+0000 7f2cd3eba700 1 -- 172.21.15.7:0/77559987 shutdown_connections 2023-09-30T01:56:37.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.637+0000 7f2cd3eba700 1 --2- 172.21.15.7:0/77559987 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2cb80357e0 0x7f2cb8037c90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:37.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.637+0000 7f2cd3eba700 1 --2- 172.21.15.7:0/77559987 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2ccc0a2390 0x7f2ccc09c410 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:37.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.637+0000 7f2cd3eba700 1 --2- 172.21.15.7:0/77559987 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2ccc096a50 0x7f2ccc09b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:37.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.637+0000 7f2cd3eba700 1 --2- 172.21.15.7:0/77559987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ccc097530 0x7f2ccc09bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:37.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.637+0000 7f2cd3eba700 1 -- 172.21.15.7:0/77559987 >> 172.21.15.7:0/77559987 conn(0x7f2ccc070cf0 msgr2=0x7f2ccc0a44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:37.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.637+0000 7f2cd3eba700 1 -- 172.21.15.7:0/77559987 shutdown_connections 2023-09-30T01:56:37.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:37.637+0000 7f2cd3eba700 1 -- 172.21.15.7:0/77559987 wait complete. 2023-09-30T01:56:40.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:56:40.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.778+0000 7f5f7e709700 10 monclient: get_monmap_and_config 2023-09-30T01:56:40.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.778+0000 7f5f7e709700 10 monclient: build_initial_monmap 2023-09-30T01:56:40.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.778+0000 7f5f7e709700 10 monclient: monmap: 2023-09-30T01:56:40.778 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:56:40.779525+0000 2023-09-30T01:56:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:56:40.779525+0000 2023-09-30T01:56:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:56:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:56:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:56:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:56:40.779 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:56:40.779 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:40.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7e709700 1 Processor -- start 2023-09-30T01:56:40.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7e709700 1 -- start start 2023-09-30T01:56:40.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7e709700 10 monclient: init 2023-09-30T01:56:40.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7e709700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:40.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7e709700 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-09-30T01:56:40.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7e709700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5f78096a50 0x7f5f78132750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:40.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7e709700 10 monclient(hunting): picked mon.noname-b con 0x7f5f78096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:40.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7e709700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5f78096e20 0x7f5f78097210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:40.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7e709700 10 monclient(hunting): picked mon.noname-c con 0x7f5f78096e20 addr v2:172.21.15.50:3300/0 2023-09-30T01:56:40.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7e709700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5f7812ef00 0x7f5f7812f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:40.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7e709700 10 monclient(hunting): picked mon.noname-a con 0x7f5f7812ef00 addr v2:172.21.15.7:3300/0 2023-09-30T01:56:40.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7e709700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:40.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7e709700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f78099480 con 0x7f5f7812ef00 2023-09-30T01:56:40.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7e709700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:40.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7e709700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f78097bf0 con 0x7f5f78096e20 2023-09-30T01:56:40.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7e709700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:40.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7e709700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f780991a0 con 0x7f5f78096a50 2023-09-30T01:56:40.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7e709700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:40.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7e709700 10 monclient(hunting): authenticate will time out at 1289.844778s 2023-09-30T01:56:40.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f77fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5f78096a50 0x7f5f78132750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:40.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7cca6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5f7812ef00 0x7f5f7812f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:40.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7cca6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5f7812ef00 0x7f5f7812f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39892/0 (socket says 172.21.15.7:39892) 2023-09-30T01:56:40.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f77fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5f78096a50 0x7f5f78132750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:58462/0 (socket says 172.21.15.7:58462) 2023-09-30T01:56:40.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7cca6700 1 -- 172.21.15.7:0/1806685638 learned_addr learned my addr 172.21.15.7:0/1806685638 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:40.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7cca6700 10 monclient(hunting): get_auth_request con 0x7f5f7812ef00 auth_method 0 2023-09-30T01:56:40.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7cca6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:40.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7cca6700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:40.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f7cca6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:40.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f77fff700 10 monclient(hunting): get_auth_request con 0x7f5f78096a50 auth_method 0 2023-09-30T01:56:40.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f77fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:40.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f77fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:40.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.779+0000 7f5f77fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:40.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.780+0000 7f5f7cca6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:40.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.780+0000 7f5f7cca6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:40.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.780+0000 7f5f7cca6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:40.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.780+0000 7f5f77fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:40.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.780+0000 7f5f77fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:40.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.780+0000 7f5f77fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:40.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.780+0000 7f5f777fe700 1 --2- 172.21.15.7:0/1806685638 >> v2:172.21.15.50:3300/0 conn(0x7f5f78096e20 0x7f5f78097210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:40.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.780+0000 7f5f7cca6700 10 monclient(hunting): handle_auth_done global_id 4458 payload 274 2023-09-30T01:56:40.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.780+0000 7f5f7cca6700 1 -- 172.21.15.7:0/1806685638 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5f78096a50 msgr2=0x7f5f78132750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:40.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.780+0000 7f5f7cca6700 1 --2- 172.21.15.7:0/1806685638 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5f78096a50 0x7f5f78132750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:40.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.780+0000 7f5f7cca6700 1 -- 172.21.15.7:0/1806685638 >> v2:172.21.15.50:3300/0 conn(0x7f5f78096e20 msgr2=0x7f5f78097210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:40.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.780+0000 7f5f7cca6700 1 --2- 172.21.15.7:0/1806685638 >> v2:172.21.15.50:3300/0 conn(0x7f5f78096e20 0x7f5f78097210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:40.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.780+0000 7f5f7cca6700 10 monclient: _finish_hunting 0 2023-09-30T01:56:40.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.780+0000 7f5f7cca6700 1 monclient: found mon.noname-a 2023-09-30T01:56:40.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.780+0000 7f5f7cca6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:40.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.780+0000 7f5f7cca6700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:56:40.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.780+0000 7f5f7cca6700 1 -- 172.21.15.7:0/1806685638 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f78099760 con 0x7f5f7812ef00 2023-09-30T01:56:40.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.780+0000 7f5f777fe700 10 monclient: get_auth_request con 0x7f5f78096e20 auth_method 0 2023-09-30T01:56:40.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.780+0000 7f5f777fe700 1 --2- 172.21.15.7:0/1806685638 >> v2:172.21.15.50:3300/0 conn(0x7f5f78096e20 0x7f5f78097210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:56:40.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.780+0000 7f5f77fff700 1 --2- 172.21.15.7:0/1806685638 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5f78096a50 0x7f5f78132750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:56:40.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.782+0000 7f5f7cca6700 1 -- 172.21.15.7:0/1806685638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f7812ef00 msgr2=0x7f5f7812f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:40.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.782+0000 7f5f7cca6700 1 --2- 172.21.15.7:0/1806685638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f7812ef00 0x7f5f7812f330 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5f78071c60 tx=0x7f5f70009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9a114c79648e46 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:40.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.782+0000 7f5f76ffd700 1 -- 172.21.15.7:0/1806685638 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7000e070 con 0x7f5f7812ef00 2023-09-30T01:56:40.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.782+0000 7f5f76ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:40.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.782+0000 7f5f76ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:56:40.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.782+0000 7f5f76ffd700 10 monclient: dump: 2023-09-30T01:56:40.791 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:40.791 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:40.791 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:40.791 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:40.791 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:40.791 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:40.791 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:40.791 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:40.792 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:40.792 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:40.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.782+0000 7f5f76ffd700 10 monclient: _finish_auth 0 2023-09-30T01:56:40.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.782+0000 7f5f76ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:40.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.782+0000 7f5f7e709700 5 monclient: authenticate success, global_id 4458 2023-09-30T01:56:40.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.782+0000 7f5f76ffd700 1 -- 172.21.15.7:0/1806685638 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7000bc30 con 0x7f5f7812ef00 2023-09-30T01:56:40.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.782+0000 7f5f76ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:40.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.782+0000 7f5f7e709700 10 monclient: get_monmap_and_config success 2023-09-30T01:56:40.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.782+0000 7f5f76ffd700 1 -- 172.21.15.7:0/1806685638 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f700092e0 con 0x7f5f7812ef00 2023-09-30T01:56:40.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.782+0000 7f5f76ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:40.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.782+0000 7f5f7e709700 4 set_mon_vals no callback set 2023-09-30T01:56:40.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.782+0000 7f5f76ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:56:40.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.782+0000 7f5f76ffd700 10 monclient: dump: 2023-09-30T01:56:40.793 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:40.793 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:40.793 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:40.793 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:40.793 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:40.793 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:40.793 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:40.793 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:40.793 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:40.793 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:40.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.783+0000 7f5f7e709700 10 monclient: shutdown 2023-09-30T01:56:40.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.783+0000 7f5f7e709700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:40.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.783+0000 7f5f7e709700 1 -- 172.21.15.7:0/1806685638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f7812ef00 msgr2=0x7f5f7812f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:40.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.783+0000 7f5f7e709700 1 --2- 172.21.15.7:0/1806685638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f7812ef00 0x7f5f7812f330 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5f78071c60 tx=0x7f5f70009840 comp rx=0 tx=0).stop 2023-09-30T01:56:40.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.783+0000 7f5f7e709700 1 -- 172.21.15.7:0/1806685638 shutdown_connections 2023-09-30T01:56:40.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.783+0000 7f5f7e709700 1 --2- 172.21.15.7:0/1806685638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f7812ef00 0x7f5f7812f330 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:40.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.783+0000 7f5f7e709700 1 --2- 172.21.15.7:0/1806685638 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5f78096a50 0x7f5f78132750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:40.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.783+0000 7f5f7e709700 1 --2- 172.21.15.7:0/1806685638 >> v2:172.21.15.50:3300/0 conn(0x7f5f78096e20 0x7f5f78097210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:40.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.783+0000 7f5f7e709700 1 -- 172.21.15.7:0/1806685638 >> 172.21.15.7:0/1806685638 conn(0x7f5f78070cf0 msgr2=0x7f5f780710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:40.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.783+0000 7f5f7e709700 1 -- 172.21.15.7:0/1806685638 shutdown_connections 2023-09-30T01:56:40.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.783+0000 7f5f7e709700 1 -- 172.21.15.7:0/1806685638 wait complete. 2023-09-30T01:56:40.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.783+0000 7f5f7e709700 10 monclient: build_initial_monmap 2023-09-30T01:56:40.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.783+0000 7f5f7e709700 10 monclient: monmap: 2023-09-30T01:56:40.794 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:40.795 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:56:40.795 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:56:40.795 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:56:40.795 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:40.795 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:40.795 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:56:40.795 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:56:40.795 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:56:40.795 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:40.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7e709700 1 Processor -- start 2023-09-30T01:56:40.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7e709700 1 -- start start 2023-09-30T01:56:40.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7e709700 10 monclient: init 2023-09-30T01:56:40.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7e709700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:40.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7e709700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:56:40.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7e709700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f78096a50 0x7f5f7812dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:40.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7e709700 10 monclient(hunting): picked mon.noname-a con 0x7f5f78096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:56:40.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7e709700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f78096e20 0x7f5f780a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:40.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7e709700 10 monclient(hunting): picked mon.noname-b con 0x7f5f78096e20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:56:40.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7e709700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5f7812ef00 0x7f5f780a0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:40.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7e709700 10 monclient(hunting): picked mon.noname-c con 0x7f5f7812ef00 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:40.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7e709700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:40.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7e709700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7811e180 con 0x7f5f78096a50 2023-09-30T01:56:40.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7e709700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:40.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7e709700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7811d040 con 0x7f5f78096e20 2023-09-30T01:56:40.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7e709700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:40.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7e709700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7811dea0 con 0x7f5f7812ef00 2023-09-30T01:56:40.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7e709700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:40.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7e709700 10 monclient(hunting): authenticate will time out at 1289.849777s 2023-09-30T01:56:40.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f77fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f78096a50 0x7f5f7812dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:40.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f77fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f78096a50 0x7f5f7812dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39908/0 (socket says 172.21.15.7:39908) 2023-09-30T01:56:40.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f77fff700 1 -- 172.21.15.7:0/2255020925 learned_addr learned my addr 172.21.15.7:0/2255020925 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:40.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f77fff700 10 monclient(hunting): get_auth_request con 0x7f5f78096a50 auth_method 0 2023-09-30T01:56:40.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f77fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:40.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f77fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:40.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f77fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:40.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7cca6700 1 --2- 172.21.15.7:0/2255020925 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5f7812ef00 0x7f5f780a0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:40.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7cca6700 10 monclient(hunting): get_auth_request con 0x7f5f7812ef00 auth_method 0 2023-09-30T01:56:40.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7cca6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:40.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7cca6700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:40.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f7cca6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:40.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f77fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:40.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.784+0000 7f5f77fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:40.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.785+0000 7f5f77fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:40.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.785+0000 7f5f7cca6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:40.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.785+0000 7f5f7cca6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:40.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.785+0000 7f5f7cca6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:40.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.785+0000 7f5f777fe700 1 --2- 172.21.15.7:0/2255020925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f78096e20 0x7f5f780a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:40.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.785+0000 7f5f777fe700 10 monclient(hunting): get_auth_request con 0x7f5f78096e20 auth_method 0 2023-09-30T01:56:40.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.785+0000 7f5f777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:40.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.785+0000 7f5f777fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:40.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.785+0000 7f5f777fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:40.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.785+0000 7f5f77fff700 10 monclient(hunting): handle_auth_done global_id 4461 payload 824 2023-09-30T01:56:40.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.785+0000 7f5f77fff700 1 -- 172.21.15.7:0/2255020925 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5f7812ef00 msgr2=0x7f5f780a0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:40.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.785+0000 7f5f77fff700 1 --2- 172.21.15.7:0/2255020925 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5f7812ef00 0x7f5f780a0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:40.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.785+0000 7f5f77fff700 1 -- 172.21.15.7:0/2255020925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f78096e20 msgr2=0x7f5f780a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:40.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.785+0000 7f5f77fff700 1 --2- 172.21.15.7:0/2255020925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f78096e20 0x7f5f780a0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:40.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.785+0000 7f5f77fff700 10 monclient: _finish_hunting 0 2023-09-30T01:56:40.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.785+0000 7f5f77fff700 1 monclient: found mon.noname-a 2023-09-30T01:56:40.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.785+0000 7f5f77fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:40.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.785+0000 7f5f77fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:56:40.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.785+0000 7f5f77fff700 1 -- 172.21.15.7:0/2255020925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f60009420 con 0x7f5f78096a50 2023-09-30T01:56:40.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.785+0000 7f5f777fe700 1 --2- 172.21.15.7:0/2255020925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f78096e20 0x7f5f780a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:56:40.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f77fff700 1 -- 172.21.15.7:0/2255020925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f78096a50 msgr2=0x7f5f7812dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:40.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f77fff700 1 --2- 172.21.15.7:0/2255020925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f78096a50 0x7f5f7812dd50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5f6000ec80 tx=0x7f5f600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:40.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f6ffff700 1 -- 172.21.15.7:0/2255020925 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6000cb00 con 0x7f5f78096a50 2023-09-30T01:56:40.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f6ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:40.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f6ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:56:40.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f6ffff700 10 monclient: dump: 2023-09-30T01:56:40.804 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:40.804 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:40.805 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:40.805 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:40.805 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:40.805 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:40.805 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:40.805 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:40.805 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:40.805 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:40.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f6ffff700 10 monclient: _finish_auth 0 2023-09-30T01:56:40.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f6ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:40.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f6ffff700 1 -- 172.21.15.7:0/2255020925 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6000cc60 con 0x7f5f78096a50 2023-09-30T01:56:40.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f7e709700 5 monclient: authenticate success, global_id 4461 2023-09-30T01:56:40.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f7e709700 10 monclient: _renew_subs 2023-09-30T01:56:40.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f7e709700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:40.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f7e709700 1 -- 172.21.15.7:0/2255020925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f70009ae0 con 0x7f5f78096a50 2023-09-30T01:56:40.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f6ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:40.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f6ffff700 1 -- 172.21.15.7:0/2255020925 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f60013470 con 0x7f5f78096a50 2023-09-30T01:56:40.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f6ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:40.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f6ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:56:40.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f6ffff700 10 monclient: dump: 2023-09-30T01:56:40.807 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:40.807 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:40.807 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:40.807 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:40.807 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:40.807 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:40.807 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:40.807 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:40.807 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:40.807 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:40.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f767fc700 4 set_mon_vals no callback set 2023-09-30T01:56:40.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f7e709700 10 monclient: _renew_subs 2023-09-30T01:56:40.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f7e709700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:40.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.787+0000 7f5f7e709700 1 -- 172.21.15.7:0/2255020925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f7811c370 con 0x7f5f78096a50 2023-09-30T01:56:40.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.788+0000 7f5f6ffff700 1 -- 172.21.15.7:0/2255020925 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f5f600137e0 con 0x7f5f78096a50 2023-09-30T01:56:40.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.788+0000 7f5f7e709700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:40.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.788+0000 7f5f7e709700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:40.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.788+0000 7f5f7e709700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:40.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.788+0000 7f5f7e709700 1 -- 172.21.15.7:0/2255020925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f78064c10 con 0x7f5f78096a50 2023-09-30T01:56:40.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.788+0000 7f5f6ffff700 1 --2- 172.21.15.7:0/2255020925 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5f580357e0 0x7f5f58037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:40.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.788+0000 7f5f6ffff700 1 -- 172.21.15.7:0/2255020925 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f5f600480a0 con 0x7f5f78096a50 2023-09-30T01:56:40.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.789+0000 7f5f777fe700 1 --2- 172.21.15.7:0/2255020925 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5f580357e0 0x7f5f58037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:40.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.789+0000 7f5f777fe700 10 monclient: get_auth_request con 0x7f5f580357e0 auth_method 0 2023-09-30T01:56:40.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.791+0000 7f5f777fe700 1 --2- 172.21.15.7:0/2255020925 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5f580357e0 0x7f5f58037c90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5f7811bf70 tx=0x7f5f68006f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:40.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.793+0000 7f5f6ffff700 1 -- 172.21.15.7:0/2255020925 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5f600236b0 con 0x7f5f78096a50 2023-09-30T01:56:40.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.793+0000 7f5f6ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:40.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:40.793+0000 7f5f6ffff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:56:41.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:41.029+0000 7f5f7e709700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:56:41.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:41.029+0000 7f5f7e709700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:41.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:41.029+0000 7f5f7e709700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:41.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:41.029+0000 7f5f7e709700 1 -- 172.21.15.7:0/2255020925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5f7811dfe0 con 0x7f5f78096a50 2023-09-30T01:56:41.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:41.029+0000 7f5f6ffff700 1 -- 172.21.15.7:0/2255020925 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5f7811dfe0 con 0x7f5f78096a50 2023-09-30T01:56:41.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:41.029+0000 7f5f6ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:41.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:41.029+0000 7f5f6ffff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:56:41.030 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:56:41.031 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:56:41.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:41.031+0000 7f5f7e709700 1 -- 172.21.15.7:0/2255020925 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5f580357e0 msgr2=0x7f5f58037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:41.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:41.031+0000 7f5f7e709700 1 --2- 172.21.15.7:0/2255020925 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5f580357e0 0x7f5f58037c90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5f7811bf70 tx=0x7f5f68006f90 comp rx=0 tx=0).stop 2023-09-30T01:56:41.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:41.031+0000 7f5f7e709700 10 monclient: shutdown 2023-09-30T01:56:41.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:41.031+0000 7f5f7e709700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:41.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:41.031+0000 7f5f7e709700 1 -- 172.21.15.7:0/2255020925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f78096a50 msgr2=0x7f5f7812dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:41.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:41.031+0000 7f5f7e709700 1 --2- 172.21.15.7:0/2255020925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f78096a50 0x7f5f7812dd50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5f6000ec80 tx=0x7f5f600091a0 comp rx=0 tx=0).stop 2023-09-30T01:56:41.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:41.031+0000 7f5f7e709700 1 -- 172.21.15.7:0/2255020925 shutdown_connections 2023-09-30T01:56:41.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:41.031+0000 7f5f7e709700 1 --2- 172.21.15.7:0/2255020925 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5f580357e0 0x7f5f58037c90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:41.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:41.031+0000 7f5f7e709700 1 --2- 172.21.15.7:0/2255020925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f78096a50 0x7f5f7812dd50 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:41.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:41.031+0000 7f5f7e709700 1 --2- 172.21.15.7:0/2255020925 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5f7812ef00 0x7f5f780a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:41.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:41.031+0000 7f5f7e709700 1 --2- 172.21.15.7:0/2255020925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f78096e20 0x7f5f780a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:41.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:41.031+0000 7f5f7e709700 1 -- 172.21.15.7:0/2255020925 >> 172.21.15.7:0/2255020925 conn(0x7f5f78070cf0 msgr2=0x7f5f781316a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:41.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:41.031+0000 7f5f7e709700 1 -- 172.21.15.7:0/2255020925 shutdown_connections 2023-09-30T01:56:41.035 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:41.031+0000 7f5f7e709700 1 -- 172.21.15.7:0/2255020925 wait complete. 2023-09-30T01:56:44.045 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:56:44.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.176+0000 7f66148e0700 10 monclient: get_monmap_and_config 2023-09-30T01:56:44.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.176+0000 7f66148e0700 10 monclient: build_initial_monmap 2023-09-30T01:56:44.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.176+0000 7f66148e0700 10 monclient: monmap: 2023-09-30T01:56:44.177 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:44.177 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:44.177 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:56:44.177927+0000 2023-09-30T01:56:44.177 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:56:44.177927+0000 2023-09-30T01:56:44.177 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:44.177 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:44.178 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:56:44.178 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:56:44.178 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:56:44.178 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:56:44.178 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:56:44.178 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:44.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.177+0000 7f66148e0700 1 Processor -- start 2023-09-30T01:56:44.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.177+0000 7f66148e0700 1 -- start start 2023-09-30T01:56:44.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.177+0000 7f66148e0700 10 monclient: init 2023-09-30T01:56:44.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.177+0000 7f66148e0700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:44.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.177+0000 7f66148e0700 10 monclient: _add_conns ranks=[2,0,3,4,1] 2023-09-30T01:56:44.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f66148e0700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6610098540 0x7f66101329e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:44.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f66148e0700 10 monclient(hunting): picked mon.noname-b con 0x7f6610098540 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:44.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f66148e0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6610098910 addr v1:172.21.15.7:6789/0 2023-09-30T01:56:44.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f66148e0700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6610098ce0 0x7f661012f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:44.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f66148e0700 10 monclient(hunting): picked mon.noname-a con 0x7f6610098ce0 addr v2:172.21.15.7:3300/0 2023-09-30T01:56:44.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f66148e0700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6610069f10 con 0x7f6610098910 2023-09-30T01:56:44.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f66148e0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:44.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f66148e0700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66100a0f40 con 0x7f6610098ce0 2023-09-30T01:56:44.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f66148e0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:44.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f66148e0700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6610099160 con 0x7f6610098540 2023-09-30T01:56:44.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f66148e0700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:44.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f66148e0700 10 monclient(hunting): authenticate will time out at 1293.243075s 2023-09-30T01:56:44.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660ed9d700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6610098540 0x7f66101329e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:44.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660f59e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6610098ce0 0x7f661012f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:44.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660e59c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6610098910 0x7f661012ebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51544/0 (socket says 172.21.15.7:51544) 2023-09-30T01:56:44.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660ed9d700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6610098540 0x7f66101329e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38192/0 (socket says 172.21.15.7:38192) 2023-09-30T01:56:44.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660e59c700 1 -- 172.21.15.7:0/1183545577 learned_addr learned my addr 172.21.15.7:0/1183545577 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:56:44.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660f59e700 10 monclient(hunting): get_auth_request con 0x7f6610098ce0 auth_method 0 2023-09-30T01:56:44.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660f59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:44.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660f59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:44.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660f59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:44.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660ed9d700 10 monclient(hunting): get_auth_request con 0x7f6610098540 auth_method 0 2023-09-30T01:56:44.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:44.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:44.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:44.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660f59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:44.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660f59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:44.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660f59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:44.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:44.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:44.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:44.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660ed9d700 10 monclient(hunting): handle_auth_done global_id 4475 payload 274 2023-09-30T01:56:44.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660ed9d700 1 -- 172.21.15.7:0/1183545577 >> v2:172.21.15.7:3300/0 conn(0x7f6610098ce0 msgr2=0x7f661012f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:44.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660ed9d700 1 --2- 172.21.15.7:0/1183545577 >> v2:172.21.15.7:3300/0 conn(0x7f6610098ce0 0x7f661012f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:44.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660ed9d700 1 -- 172.21.15.7:0/1183545577 >> v1:172.21.15.7:6789/0 conn(0x7f6610098910 legacy=0x7f661012ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:44.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660ed9d700 10 monclient: _finish_hunting 0 2023-09-30T01:56:44.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660ed9d700 1 monclient: found mon.noname-b 2023-09-30T01:56:44.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:44.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660ed9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:56:44.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660ed9d700 1 -- 172.21.15.7:0/1183545577 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6610096a50 con 0x7f6610098540 2023-09-30T01:56:44.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.178+0000 7f660f59e700 1 --2- 172.21.15.7:0/1183545577 >> v2:172.21.15.7:3300/0 conn(0x7f6610098ce0 0x7f661012f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:56:44.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.180+0000 7f660ed9d700 1 -- 172.21.15.7:0/1183545577 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6610098540 msgr2=0x7f66101329e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:44.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.180+0000 7f660ed9d700 1 --2- 172.21.15.7:0/1183545577 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6610098540 0x7f66101329e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f66000098e0 tx=0x7f66000091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6907c7d157bb1c5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:44.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.180+0000 7f660dd9b700 1 -- 172.21.15.7:0/1183545577 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6600011840 con 0x7f6610098540 2023-09-30T01:56:44.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.180+0000 7f660dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:44.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.180+0000 7f660dd9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:56:44.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.180+0000 7f660dd9b700 10 monclient: dump: 2023-09-30T01:56:44.186 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:44.186 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:44.186 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:44.187 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:44.187 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:44.187 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:44.187 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:44.187 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:44.187 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:44.187 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:44.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.180+0000 7f660dd9b700 10 monclient: _finish_auth 0 2023-09-30T01:56:44.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.180+0000 7f660dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:44.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.180+0000 7f660dd9b700 1 -- 172.21.15.7:0/1183545577 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66000119a0 con 0x7f6610098540 2023-09-30T01:56:44.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.180+0000 7f66148e0700 5 monclient: authenticate success, global_id 4475 2023-09-30T01:56:44.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.180+0000 7f66148e0700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:56:44.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.180+0000 7f660dd9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:44.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.181+0000 7f66148e0700 10 monclient: get_monmap_and_config success 2023-09-30T01:56:44.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.181+0000 7f660dd9b700 1 -- 172.21.15.7:0/1183545577 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6600011d40 con 0x7f6610098540 2023-09-30T01:56:44.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.181+0000 7f660dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:44.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.181+0000 7f66148e0700 4 set_mon_vals no callback set 2023-09-30T01:56:44.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.181+0000 7f660dd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:56:44.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.181+0000 7f660dd9b700 10 monclient: dump: 2023-09-30T01:56:44.188 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:44.188 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:44.188 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:44.188 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:44.188 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:44.189 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:44.189 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:44.189 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:44.189 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:44.189 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:44.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.181+0000 7f66148e0700 10 monclient: shutdown 2023-09-30T01:56:44.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.181+0000 7f66148e0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:44.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.181+0000 7f66148e0700 1 -- 172.21.15.7:0/1183545577 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6610098540 msgr2=0x7f66101329e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:44.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.181+0000 7f66148e0700 1 --2- 172.21.15.7:0/1183545577 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6610098540 0x7f66101329e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f66000098e0 tx=0x7f66000091a0 comp rx=0 tx=0).stop 2023-09-30T01:56:44.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.181+0000 7f66148e0700 1 -- 172.21.15.7:0/1183545577 shutdown_connections 2023-09-30T01:56:44.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.181+0000 7f66148e0700 1 --2- 172.21.15.7:0/1183545577 >> v2:172.21.15.7:3300/0 conn(0x7f6610098ce0 0x7f661012f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:44.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.181+0000 7f66148e0700 1 --2- 172.21.15.7:0/1183545577 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6610098540 0x7f66101329e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:44.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.181+0000 7f66148e0700 1 -- 172.21.15.7:0/1183545577 >> 172.21.15.7:0/1183545577 conn(0x7f6610071830 msgr2=0x7f66100a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:44.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.181+0000 7f66148e0700 1 -- 172.21.15.7:0/1183545577 shutdown_connections 2023-09-30T01:56:44.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.181+0000 7f66148e0700 1 -- 172.21.15.7:0/1183545577 wait complete. 2023-09-30T01:56:44.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.182+0000 7f66148e0700 10 monclient: build_initial_monmap 2023-09-30T01:56:44.190 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.182+0000 7f66148e0700 10 monclient: monmap: 2023-09-30T01:56:44.190 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:44.190 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:56:44.190 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:56:44.190 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:56:44.190 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:44.191 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:44.191 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:56:44.191 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:56:44.191 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:56:44.191 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:44.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.182+0000 7f66148e0700 1 Processor -- start 2023-09-30T01:56:44.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.182+0000 7f66148e0700 1 -- start start 2023-09-30T01:56:44.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.182+0000 7f66148e0700 10 monclient: init 2023-09-30T01:56:44.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f66148e0700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:44.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f66148e0700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:56:44.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f66148e0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6610098540 0x7f661011d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:44.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f66148e0700 10 monclient(hunting): picked mon.noname-a con 0x7f6610098540 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:56:44.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f66148e0700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6610098910 0x7f661011dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:44.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f66148e0700 10 monclient(hunting): picked mon.noname-c con 0x7f6610098910 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:44.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f66148e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6610098ce0 0x7f661011e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:44.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f66148e0700 10 monclient(hunting): picked mon.noname-b con 0x7f6610098ce0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:56:44.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f66148e0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:44.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f66148e0700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661011fc70 con 0x7f6610098540 2023-09-30T01:56:44.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f66148e0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:44.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f66148e0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66101327b0 con 0x7f6610098ce0 2023-09-30T01:56:44.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f66148e0700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:44.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f66148e0700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66100a4100 con 0x7f6610098910 2023-09-30T01:56:44.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f66148e0700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:44.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f66148e0700 10 monclient(hunting): authenticate will time out at 1293.248165s 2023-09-30T01:56:44.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6610098540 0x7f661011d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:44.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660ed9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6610098540 0x7f661011d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44998/0 (socket says 172.21.15.7:44998) 2023-09-30T01:56:44.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660ed9d700 1 -- 172.21.15.7:0/1090834882 learned_addr learned my addr 172.21.15.7:0/1090834882 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:44.199 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660ed9d700 10 monclient(hunting): get_auth_request con 0x7f6610098540 auth_method 0 2023-09-30T01:56:44.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:44.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:44.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:44.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660e59c700 1 --2- 172.21.15.7:0/1090834882 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6610098910 0x7f661011dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:44.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660e59c700 10 monclient(hunting): get_auth_request con 0x7f6610098910 auth_method 0 2023-09-30T01:56:44.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:44.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:44.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:44.200 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:44.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:44.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:44.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:44.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:44.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:44.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660ed9d700 10 monclient(hunting): handle_auth_done global_id 4467 payload 824 2023-09-30T01:56:44.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660ed9d700 1 -- 172.21.15.7:0/1090834882 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6610098910 msgr2=0x7f661011dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:44.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660ed9d700 1 --2- 172.21.15.7:0/1090834882 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6610098910 0x7f661011dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:44.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660ed9d700 1 -- 172.21.15.7:0/1090834882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6610098ce0 msgr2=0x7f661011e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:44.201 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660ed9d700 1 --2- 172.21.15.7:0/1090834882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6610098ce0 0x7f661011e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:44.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660ed9d700 10 monclient: _finish_hunting 0 2023-09-30T01:56:44.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660ed9d700 1 monclient: found mon.noname-a 2023-09-30T01:56:44.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:44.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660ed9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:56:44.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.183+0000 7f660ed9d700 1 -- 172.21.15.7:0/1090834882 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6608009a20 con 0x7f6610098540 2023-09-30T01:56:44.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.185+0000 7f660ed9d700 1 -- 172.21.15.7:0/1090834882 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6610098540 msgr2=0x7f661011d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:44.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.185+0000 7f660ed9d700 1 --2- 172.21.15.7:0/1090834882 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6610098540 0x7f661011d670 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f66000093f0 tx=0x7f6600016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:44.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.185+0000 7f65f6ffd700 1 -- 172.21.15.7:0/1090834882 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6600006af0 con 0x7f6610098540 2023-09-30T01:56:44.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.185+0000 7f65f6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:44.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.185+0000 7f65f6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:56:44.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.185+0000 7f65f6ffd700 10 monclient: dump: 2023-09-30T01:56:44.203 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:44.203 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:44.203 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:44.203 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:44.203 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:44.203 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:44.203 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:44.203 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:44.203 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:44.204 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:44.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.185+0000 7f65f6ffd700 10 monclient: _finish_auth 0 2023-09-30T01:56:44.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.185+0000 7f65f6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:44.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.185+0000 7f65f6ffd700 1 -- 172.21.15.7:0/1090834882 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6600020070 con 0x7f6610098540 2023-09-30T01:56:44.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.185+0000 7f66148e0700 5 monclient: authenticate success, global_id 4467 2023-09-30T01:56:44.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.186+0000 7f65f6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:44.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.186+0000 7f66148e0700 10 monclient: _renew_subs 2023-09-30T01:56:44.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.186+0000 7f65f6ffd700 1 -- 172.21.15.7:0/1090834882 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660000bdd0 con 0x7f6610098540 2023-09-30T01:56:44.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.186+0000 7f66148e0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:44.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.186+0000 7f66148e0700 1 -- 172.21.15.7:0/1090834882 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6600009420 con 0x7f6610098540 2023-09-30T01:56:44.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.186+0000 7f65f6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:44.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.186+0000 7f660d59a700 4 set_mon_vals no callback set 2023-09-30T01:56:44.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.186+0000 7f65f6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:56:44.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.186+0000 7f65f6ffd700 10 monclient: dump: 2023-09-30T01:56:44.205 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:44.205 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:44.205 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:44.205 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:44.205 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:44.206 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:44.206 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:44.206 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:44.206 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:44.206 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:44.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.186+0000 7f66148e0700 10 monclient: _renew_subs 2023-09-30T01:56:44.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.186+0000 7f66148e0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:44.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.186+0000 7f66148e0700 1 -- 172.21.15.7:0/1090834882 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66100a05c0 con 0x7f6610098540 2023-09-30T01:56:44.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.187+0000 7f65f6ffd700 1 -- 172.21.15.7:0/1090834882 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f6600022030 con 0x7f6610098540 2023-09-30T01:56:44.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.187+0000 7f66148e0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:44.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.187+0000 7f66148e0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:44.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.187+0000 7f66148e0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:44.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.187+0000 7f66148e0700 1 -- 172.21.15.7:0/1090834882 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661009df00 con 0x7f6610098540 2023-09-30T01:56:44.207 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.187+0000 7f65f6ffd700 1 --2- 172.21.15.7:0/1090834882 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f65fc035830 0x7f65fc037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:44.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.187+0000 7f65f6ffd700 1 -- 172.21.15.7:0/1090834882 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f6600049520 con 0x7f6610098540 2023-09-30T01:56:44.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.187+0000 7f660e59c700 1 --2- 172.21.15.7:0/1090834882 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f65fc035830 0x7f65fc037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:44.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.187+0000 7f660e59c700 10 monclient: get_auth_request con 0x7f65fc035830 auth_method 0 2023-09-30T01:56:44.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.190+0000 7f660e59c700 1 --2- 172.21.15.7:0/1090834882 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f65fc035830 0x7f65fc037ce0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f661011ee50 tx=0x7f6604007f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:44.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.191+0000 7f65f6ffd700 1 -- 172.21.15.7:0/1090834882 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f660001b660 con 0x7f6610098540 2023-09-30T01:56:44.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.191+0000 7f65f6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:44.208 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.191+0000 7f65f6ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:56:44.426 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.426+0000 7f66148e0700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:56:44.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.426+0000 7f66148e0700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:44.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.426+0000 7f66148e0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:44.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.426+0000 7f66148e0700 1 -- 172.21.15.7:0/1090834882 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f661009df00 con 0x7f6610098540 2023-09-30T01:56:44.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.427+0000 7f65f6ffd700 1 -- 172.21.15.7:0/1090834882 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f660001a3d0 con 0x7f6610098540 2023-09-30T01:56:44.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.427+0000 7f65f6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:44.427 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.427+0000 7f65f6ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:56:44.427 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:56:44.427 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:56:44.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.429+0000 7f66148e0700 1 -- 172.21.15.7:0/1090834882 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f65fc035830 msgr2=0x7f65fc037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:44.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.429+0000 7f66148e0700 1 --2- 172.21.15.7:0/1090834882 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f65fc035830 0x7f65fc037ce0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f661011ee50 tx=0x7f6604007f90 comp rx=0 tx=0).stop 2023-09-30T01:56:44.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.429+0000 7f66148e0700 10 monclient: shutdown 2023-09-30T01:56:44.429 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.429+0000 7f66148e0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:44.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.429+0000 7f66148e0700 1 -- 172.21.15.7:0/1090834882 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6610098540 msgr2=0x7f661011d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:44.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.429+0000 7f66148e0700 1 --2- 172.21.15.7:0/1090834882 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6610098540 0x7f661011d670 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f66000093f0 tx=0x7f6600016000 comp rx=0 tx=0).stop 2023-09-30T01:56:44.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.429+0000 7f66148e0700 1 -- 172.21.15.7:0/1090834882 shutdown_connections 2023-09-30T01:56:44.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.429+0000 7f66148e0700 1 --2- 172.21.15.7:0/1090834882 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f65fc035830 0x7f65fc037ce0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:44.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.429+0000 7f66148e0700 1 --2- 172.21.15.7:0/1090834882 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6610098540 0x7f661011d670 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:44.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.429+0000 7f66148e0700 1 --2- 172.21.15.7:0/1090834882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6610098ce0 0x7f661011e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:44.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.429+0000 7f66148e0700 1 --2- 172.21.15.7:0/1090834882 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6610098910 0x7f661011dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:44.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.429+0000 7f66148e0700 1 -- 172.21.15.7:0/1090834882 >> 172.21.15.7:0/1090834882 conn(0x7f6610071830 msgr2=0x7f66101319a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:44.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.429+0000 7f66148e0700 1 -- 172.21.15.7:0/1090834882 shutdown_connections 2023-09-30T01:56:44.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:44.429+0000 7f66148e0700 1 -- 172.21.15.7:0/1090834882 wait complete. 2023-09-30T01:56:47.441 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:56:47.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.572+0000 7f3a701b3700 10 monclient: get_monmap_and_config 2023-09-30T01:56:47.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.572+0000 7f3a701b3700 10 monclient: build_initial_monmap 2023-09-30T01:56:47.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.572+0000 7f3a701b3700 10 monclient: monmap: 2023-09-30T01:56:47.572 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:47.573 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:47.573 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:56:47.573171+0000 2023-09-30T01:56:47.573 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:56:47.573171+0000 2023-09-30T01:56:47.573 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:47.573 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:47.573 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:56:47.573 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:56:47.573 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:56:47.573 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:56:47.573 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:56:47.573 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:47.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.572+0000 7f3a701b3700 1 Processor -- start 2023-09-30T01:56:47.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.572+0000 7f3a701b3700 1 -- start start 2023-09-30T01:56:47.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.572+0000 7f3a701b3700 10 monclient: init 2023-09-30T01:56:47.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.573+0000 7f3a701b3700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:47.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.573+0000 7f3a701b3700 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-09-30T01:56:47.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.573+0000 7f3a701b3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a68096a50 addr v1:172.21.15.7:6789/0 2023-09-30T01:56:47.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.573+0000 7f3a701b3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a68096e20 addr v1:172.21.15.50:6789/0 2023-09-30T01:56:47.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.573+0000 7f3a701b3700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3a68138720 0x7f3a6813ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:47.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.573+0000 7f3a701b3700 10 monclient(hunting): picked mon.noname-c con 0x7f3a68138720 addr v2:172.21.15.50:3300/0 2023-09-30T01:56:47.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.573+0000 7f3a701b3700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a68099440 con 0x7f3a68096a50 2023-09-30T01:56:47.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.573+0000 7f3a701b3700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a68132d90 con 0x7f3a68096e20 2023-09-30T01:56:47.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.573+0000 7f3a701b3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:47.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.573+0000 7f3a701b3700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a68097b40 con 0x7f3a68138720 2023-09-30T01:56:47.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.573+0000 7f3a701b3700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:47.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.573+0000 7f3a701b3700 10 monclient(hunting): authenticate will time out at 1296.638513s 2023-09-30T01:56:47.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.573+0000 7f3a6df4f700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3a68096a50 0x7f3a68132680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51548/0 (socket says 172.21.15.7:51548) 2023-09-30T01:56:47.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.573+0000 7f3a6df4f700 1 -- 172.21.15.7:0/3833903385 learned_addr learned my addr 172.21.15.7:0/3833903385 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:56:47.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.573+0000 7f3a6e750700 1 --2- 172.21.15.7:0/3833903385 >> v2:172.21.15.50:3300/0 conn(0x7f3a68138720 0x7f3a6813ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:47.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.573+0000 7f3a6e750700 10 monclient(hunting): get_auth_request con 0x7f3a68138720 auth_method 0 2023-09-30T01:56:47.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.573+0000 7f3a6e750700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:47.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.573+0000 7f3a6e750700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:47.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.573+0000 7f3a6e750700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:47.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.574+0000 7f3a6e750700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:47.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.574+0000 7f3a6e750700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:47.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.574+0000 7f3a6e750700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:47.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.574+0000 7f3a6e750700 10 monclient(hunting): handle_auth_done global_id 4429 payload 274 2023-09-30T01:56:47.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.574+0000 7f3a6e750700 1 -- 172.21.15.7:0/3833903385 >> v1:172.21.15.50:6789/0 conn(0x7f3a68096e20 legacy=0x7f3a68097220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:47.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.574+0000 7f3a6e750700 1 -- 172.21.15.7:0/3833903385 >> v1:172.21.15.7:6789/0 conn(0x7f3a68096a50 legacy=0x7f3a68132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:47.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.574+0000 7f3a6e750700 10 monclient: _finish_hunting 0 2023-09-30T01:56:47.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.574+0000 7f3a6e750700 1 monclient: found mon.noname-c 2023-09-30T01:56:47.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.574+0000 7f3a6e750700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:47.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.574+0000 7f3a6e750700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3300/0 2023-09-30T01:56:47.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.574+0000 7f3a6e750700 1 -- 172.21.15.7:0/3833903385 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a68099160 con 0x7f3a68138720 2023-09-30T01:56:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.576+0000 7f3a6e750700 1 -- 172.21.15.7:0/3833903385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a68138720 msgr2=0x7f3a6813ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.576+0000 7f3a6e750700 1 --2- 172.21.15.7:0/3833903385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a68138720 0x7f3a6813ab00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3a58009e60 tx=0x7f3a58009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dab6e834d1b9c99 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.576+0000 7f3a6cf4d700 1 -- 172.21.15.7:0/3833903385 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5800e070 con 0x7f3a68138720 2023-09-30T01:56:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.576+0000 7f3a6cf4d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.576+0000 7f3a6cf4d700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:56:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.576+0000 7f3a6cf4d700 10 monclient: dump: 2023-09-30T01:56:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:47.584 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:47.585 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.576+0000 7f3a6cf4d700 10 monclient: _finish_auth 0 2023-09-30T01:56:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.576+0000 7f3a6cf4d700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.576+0000 7f3a6cf4d700 1 -- 172.21.15.7:0/3833903385 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a58012430 con 0x7f3a68138720 2023-09-30T01:56:47.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.576+0000 7f3a701b3700 5 monclient: authenticate success, global_id 4429 2023-09-30T01:56:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.576+0000 7f3a701b3700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:56:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.576+0000 7f3a6cf4d700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.576+0000 7f3a701b3700 10 monclient: get_monmap_and_config success 2023-09-30T01:56:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.576+0000 7f3a6cf4d700 1 -- 172.21.15.7:0/3833903385 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5800bea0 con 0x7f3a68138720 2023-09-30T01:56:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.576+0000 7f3a6cf4d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.576+0000 7f3a701b3700 4 set_mon_vals no callback set 2023-09-30T01:56:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.576+0000 7f3a6cf4d700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:56:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.576+0000 7f3a6cf4d700 10 monclient: dump: 2023-09-30T01:56:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:47.586 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:47.587 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.577+0000 7f3a701b3700 10 monclient: shutdown 2023-09-30T01:56:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.577+0000 7f3a701b3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.577+0000 7f3a701b3700 1 -- 172.21.15.7:0/3833903385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a68138720 msgr2=0x7f3a6813ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.577+0000 7f3a701b3700 1 --2- 172.21.15.7:0/3833903385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a68138720 0x7f3a6813ab00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3a58009e60 tx=0x7f3a58009780 comp rx=0 tx=0).stop 2023-09-30T01:56:47.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.577+0000 7f3a701b3700 1 -- 172.21.15.7:0/3833903385 shutdown_connections 2023-09-30T01:56:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.577+0000 7f3a701b3700 1 --2- 172.21.15.7:0/3833903385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a68138720 0x7f3a6813ab00 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.577+0000 7f3a701b3700 1 -- 172.21.15.7:0/3833903385 >> 172.21.15.7:0/3833903385 conn(0x7f3a68070cf0 msgr2=0x7f3a680710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.577+0000 7f3a701b3700 1 -- 172.21.15.7:0/3833903385 shutdown_connections 2023-09-30T01:56:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.577+0000 7f3a701b3700 1 -- 172.21.15.7:0/3833903385 wait complete. 2023-09-30T01:56:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.577+0000 7f3a701b3700 10 monclient: build_initial_monmap 2023-09-30T01:56:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.577+0000 7f3a701b3700 10 monclient: monmap: 2023-09-30T01:56:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:56:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:56:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:56:47.588 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:47.589 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:47.589 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:56:47.589 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:56:47.589 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:56:47.589 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:47.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a701b3700 1 Processor -- start 2023-09-30T01:56:47.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a701b3700 1 -- start start 2023-09-30T01:56:47.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a701b3700 10 monclient: init 2023-09-30T01:56:47.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a701b3700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:47.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a701b3700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:56:47.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a701b3700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3a68096a50 0x7f3a6809a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:47.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a701b3700 10 monclient(hunting): picked mon.noname-c con 0x7f3a68096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:47.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a701b3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a68096e20 0x7f3a6809aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:47.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a701b3700 10 monclient(hunting): picked mon.noname-b con 0x7f3a68096e20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:56:47.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a701b3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a68138720 0x7f3a6809afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:47.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a701b3700 10 monclient(hunting): picked mon.noname-a con 0x7f3a68138720 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:56:47.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a701b3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:47.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a701b3700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a680a11b0 con 0x7f3a68138720 2023-09-30T01:56:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a701b3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a701b3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6809dcb0 con 0x7f3a68096e20 2023-09-30T01:56:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a701b3700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a701b3700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6809df90 con 0x7f3a68096a50 2023-09-30T01:56:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a701b3700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a701b3700 10 monclient(hunting): authenticate will time out at 1296.643812s 2023-09-30T01:56:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a6e750700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a68138720 0x7f3a6809afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a6df4f700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3a68096a50 0x7f3a6809a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a6e750700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a68138720 0x7f3a6809afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45012/0 (socket says 172.21.15.7:45012) 2023-09-30T01:56:47.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a6e750700 1 -- 172.21.15.7:0/1903913228 learned_addr learned my addr 172.21.15.7:0/1903913228 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a6df4f700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3a68096a50 0x7f3a6809a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38208/0 (socket says 172.21.15.7:38208) 2023-09-30T01:56:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a6e750700 10 monclient(hunting): get_auth_request con 0x7f3a68138720 auth_method 0 2023-09-30T01:56:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a6e750700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a6e750700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a6e750700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a6df4f700 10 monclient(hunting): get_auth_request con 0x7f3a68096a50 auth_method 0 2023-09-30T01:56:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a6df4f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a6df4f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a6df4f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.578+0000 7f3a6d74e700 1 --2- 172.21.15.7:0/1903913228 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a68096e20 0x7f3a6809aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6d74e700 10 monclient(hunting): get_auth_request con 0x7f3a68096e20 auth_method 0 2023-09-30T01:56:47.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6d74e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6d74e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6d74e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6e750700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6e750700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6e750700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6df4f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6df4f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6df4f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6d74e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6d74e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:47.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6d74e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6e750700 10 monclient(hunting): handle_auth_done global_id 4470 payload 824 2023-09-30T01:56:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6e750700 1 -- 172.21.15.7:0/1903913228 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3a68096a50 msgr2=0x7f3a6809a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6e750700 1 --2- 172.21.15.7:0/1903913228 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3a68096a50 0x7f3a6809a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6e750700 1 -- 172.21.15.7:0/1903913228 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a68096e20 msgr2=0x7f3a6809aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6e750700 1 --2- 172.21.15.7:0/1903913228 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a68096e20 0x7f3a6809aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6e750700 10 monclient: _finish_hunting 0 2023-09-30T01:56:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6e750700 1 monclient: found mon.noname-a 2023-09-30T01:56:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6e750700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6e750700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:56:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6e750700 1 -- 172.21.15.7:0/1903913228 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a58009a20 con 0x7f3a68138720 2023-09-30T01:56:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.579+0000 7f3a6df4f700 1 --2- 172.21.15.7:0/1903913228 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3a68096a50 0x7f3a6809a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:56:47.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.581+0000 7f3a6e750700 1 -- 172.21.15.7:0/1903913228 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a68138720 msgr2=0x7f3a6809afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:47.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.581+0000 7f3a6e750700 1 --2- 172.21.15.7:0/1903913228 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a68138720 0x7f3a6809afd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3a680995d0 tx=0x7f3a58018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:47.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.581+0000 7f3a65ffb700 1 -- 172.21.15.7:0/1903913228 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58026430 con 0x7f3a68138720 2023-09-30T01:56:47.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.581+0000 7f3a65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:47.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.581+0000 7f3a65ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:56:47.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.581+0000 7f3a65ffb700 10 monclient: dump: 2023-09-30T01:56:47.595 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:47.595 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:47.595 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:47.595 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:47.595 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:47.595 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:47.595 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:47.596 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:47.596 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:47.596 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:47.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.581+0000 7f3a65ffb700 10 monclient: _finish_auth 0 2023-09-30T01:56:47.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.581+0000 7f3a65ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:47.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.581+0000 7f3a701b3700 5 monclient: authenticate success, global_id 4470 2023-09-30T01:56:47.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.581+0000 7f3a65ffb700 1 -- 172.21.15.7:0/1903913228 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a58021070 con 0x7f3a68138720 2023-09-30T01:56:47.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.581+0000 7f3a701b3700 10 monclient: _renew_subs 2023-09-30T01:56:47.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.581+0000 7f3a701b3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:47.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.581+0000 7f3a701b3700 1 -- 172.21.15.7:0/1903913228 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6809c180 con 0x7f3a68138720 2023-09-30T01:56:47.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.581+0000 7f3a65ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:47.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.581+0000 7f3a65ffb700 1 -- 172.21.15.7:0/1903913228 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a580267e0 con 0x7f3a68138720 2023-09-30T01:56:47.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.581+0000 7f3a65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:47.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.581+0000 7f3a65ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:56:47.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.581+0000 7f3a65ffb700 10 monclient: dump: 2023-09-30T01:56:47.597 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:47.597 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:47.597 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:47.597 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:47.597 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:47.597 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:47.597 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:47.597 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:47.597 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:47.597 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:47.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.581+0000 7f3a701b3700 10 monclient: _renew_subs 2023-09-30T01:56:47.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.581+0000 7f3a701b3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:47.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.582+0000 7f3a701b3700 1 -- 172.21.15.7:0/1903913228 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a6809cdc0 con 0x7f3a68138720 2023-09-30T01:56:47.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.581+0000 7f3a67fff700 4 set_mon_vals no callback set 2023-09-30T01:56:47.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.582+0000 7f3a701b3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:47.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.582+0000 7f3a701b3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:47.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.582+0000 7f3a701b3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:47.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.582+0000 7f3a65ffb700 1 -- 172.21.15.7:0/1903913228 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f3a58023030 con 0x7f3a68138720 2023-09-30T01:56:47.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.582+0000 7f3a701b3700 1 -- 172.21.15.7:0/1903913228 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6809d070 con 0x7f3a68138720 2023-09-30T01:56:47.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.583+0000 7f3a65ffb700 1 --2- 172.21.15.7:0/1903913228 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3a540357e0 0x7f3a54037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:47.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.583+0000 7f3a65ffb700 1 -- 172.21.15.7:0/1903913228 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f3a5804a000 con 0x7f3a68138720 2023-09-30T01:56:47.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.583+0000 7f3a6df4f700 1 --2- 172.21.15.7:0/1903913228 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3a540357e0 0x7f3a54037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:47.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.583+0000 7f3a6df4f700 10 monclient: get_auth_request con 0x7f3a540357e0 auth_method 0 2023-09-30T01:56:47.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.585+0000 7f3a6df4f700 1 --2- 172.21.15.7:0/1903913228 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3a540357e0 0x7f3a54037c90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009ed0 tx=0x7f3a60009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:47.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.587+0000 7f3a65ffb700 1 -- 172.21.15.7:0/1903913228 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3a5801b990 con 0x7f3a68138720 2023-09-30T01:56:47.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.587+0000 7f3a65ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:47.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.587+0000 7f3a65ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:56:47.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.822+0000 7f3a701b3700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:56:47.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.822+0000 7f3a701b3700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:47.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.822+0000 7f3a701b3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:56:47.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.822+0000 7f3a701b3700 1 -- 172.21.15.7:0/1903913228 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3a6809d070 con 0x7f3a68138720 2023-09-30T01:56:47.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.823+0000 7f3a65ffb700 1 -- 172.21.15.7:0/1903913228 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3a5801b3a0 con 0x7f3a68138720 2023-09-30T01:56:47.823 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.823+0000 7f3a65ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:47.824 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.823+0000 7f3a65ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:56:47.824 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:56:47.824 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:56:47.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.825+0000 7f3a701b3700 1 -- 172.21.15.7:0/1903913228 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3a540357e0 msgr2=0x7f3a54037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:47.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.825+0000 7f3a701b3700 1 --2- 172.21.15.7:0/1903913228 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3a540357e0 0x7f3a54037c90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009ed0 tx=0x7f3a60009060 comp rx=0 tx=0).stop 2023-09-30T01:56:47.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.825+0000 7f3a701b3700 10 monclient: shutdown 2023-09-30T01:56:47.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.825+0000 7f3a701b3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:47.825 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.825+0000 7f3a701b3700 1 -- 172.21.15.7:0/1903913228 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a68138720 msgr2=0x7f3a6809afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:47.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.825+0000 7f3a701b3700 1 --2- 172.21.15.7:0/1903913228 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a68138720 0x7f3a6809afd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3a680995d0 tx=0x7f3a58018000 comp rx=0 tx=0).stop 2023-09-30T01:56:47.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.825+0000 7f3a701b3700 1 -- 172.21.15.7:0/1903913228 shutdown_connections 2023-09-30T01:56:47.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.825+0000 7f3a701b3700 1 --2- 172.21.15.7:0/1903913228 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3a540357e0 0x7f3a54037c90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:47.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.825+0000 7f3a701b3700 1 --2- 172.21.15.7:0/1903913228 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a68138720 0x7f3a6809afd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:47.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.825+0000 7f3a701b3700 1 --2- 172.21.15.7:0/1903913228 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3a68096a50 0x7f3a6809a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:47.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.825+0000 7f3a701b3700 1 --2- 172.21.15.7:0/1903913228 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3a68096e20 0x7f3a6809aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:47.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.825+0000 7f3a701b3700 1 -- 172.21.15.7:0/1903913228 >> 172.21.15.7:0/1903913228 conn(0x7f3a68070cf0 msgr2=0x7f3a68138af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:47.826 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.825+0000 7f3a701b3700 1 -- 172.21.15.7:0/1903913228 shutdown_connections 2023-09-30T01:56:47.827 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:47.825+0000 7f3a701b3700 1 -- 172.21.15.7:0/1903913228 wait complete. 2023-09-30T01:56:50.837 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:56:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.969+0000 7f31bbb18700 10 monclient: get_monmap_and_config 2023-09-30T01:56:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.969+0000 7f31bbb18700 10 monclient: build_initial_monmap 2023-09-30T01:56:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.969+0000 7f31bbb18700 10 monclient: monmap: 2023-09-30T01:56:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:50.968 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:56:50.969323+0000 2023-09-30T01:56:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:56:50.969323+0000 2023-09-30T01:56:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:56:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:56:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:56:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:56:50.969 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:56:50.969 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.969+0000 7f31bbb18700 1 Processor -- start 2023-09-30T01:56:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31bbb18700 1 -- start start 2023-09-30T01:56:50.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31bbb18700 10 monclient: init 2023-09-30T01:56:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31bbb18700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31bbb18700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-09-30T01:56:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31bbb18700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31b40a29e0 addr v1:172.21.15.7:6789/0 2023-09-30T01:56:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31bbb18700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f31b4097160 0x7f31b4137550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31bbb18700 10 monclient(hunting): picked mon.noname-b con 0x7f31b4097160 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31bbb18700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f31b4097530 addr v1:172.21.15.50:6789/0 2023-09-30T01:56:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31bbb18700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b4069cc0 con 0x7f31b40a29e0 2023-09-30T01:56:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31bbb18700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b40a2db0 con 0x7f31b4097530 2023-09-30T01:56:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31bbb18700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:50.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31bbb18700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b4098ea0 con 0x7f31b4097160 2023-09-30T01:56:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31bbb18700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31bbb18700 10 monclient(hunting): authenticate will time out at 1300.034589s 2023-09-30T01:56:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31b98b4700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f31b40a29e0 0x7f31b4096a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:51552/0 (socket says 172.21.15.7:51552) 2023-09-30T01:56:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31b98b4700 1 -- 172.21.15.7:0/980349051 learned_addr learned my addr 172.21.15.7:0/980349051 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:56:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31b90b3700 1 --2- 172.21.15.7:0/980349051 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f31b4097160 0x7f31b4137550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31b90b3700 10 monclient(hunting): get_auth_request con 0x7f31b4097160 auth_method 0 2023-09-30T01:56:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31b90b3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:50.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31b90b3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:50.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31b90b3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:50.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31b90b3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:50.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31b90b3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:50.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31b90b3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:50.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.970+0000 7f31b90b3700 10 monclient(hunting): handle_auth_done global_id 4484 payload 274 2023-09-30T01:56:50.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.971+0000 7f31b90b3700 1 -- 172.21.15.7:0/980349051 >> v1:172.21.15.50:6789/0 conn(0x7f31b4097530 legacy=0x7f31b4139aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:50.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.971+0000 7f31b90b3700 1 -- 172.21.15.7:0/980349051 >> v1:172.21.15.7:6789/0 conn(0x7f31b40a29e0 legacy=0x7f31b4096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:50.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.971+0000 7f31b90b3700 10 monclient: _finish_hunting 0 2023-09-30T01:56:50.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.971+0000 7f31b90b3700 1 monclient: found mon.noname-b 2023-09-30T01:56:50.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.971+0000 7f31b90b3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:50.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.971+0000 7f31b90b3700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:56:50.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.971+0000 7f31b90b3700 1 -- 172.21.15.7:0/980349051 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b4098270 con 0x7f31b4097160 2023-09-30T01:56:50.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.973+0000 7f31b90b3700 1 -- 172.21.15.7:0/980349051 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f31b4097160 msgr2=0x7f31b4137550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:50.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.973+0000 7f31b90b3700 1 --2- 172.21.15.7:0/980349051 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f31b4097160 0x7f31b4137550 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f31a8009910 tx=0x7f31a80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e93bafc1763493cf server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:50.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.973+0000 7f31b88b2700 1 -- 172.21.15.7:0/980349051 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a800ba30 con 0x7f31b4097160 2023-09-30T01:56:50.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.973+0000 7f31b88b2700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:50.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.973+0000 7f31b88b2700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:56:50.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.973+0000 7f31b88b2700 10 monclient: dump: 2023-09-30T01:56:50.982 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:50.983 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:50.983 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:50.983 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:50.983 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:50.983 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:50.983 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:50.983 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:50.983 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:50.983 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:50.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.973+0000 7f31b88b2700 10 monclient: _finish_auth 0 2023-09-30T01:56:50.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.973+0000 7f31b88b2700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:50.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.973+0000 7f31b88b2700 1 -- 172.21.15.7:0/980349051 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a800bb90 con 0x7f31b4097160 2023-09-30T01:56:50.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.973+0000 7f31b88b2700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:50.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.973+0000 7f31b88b2700 1 -- 172.21.15.7:0/980349051 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a8011420 con 0x7f31b4097160 2023-09-30T01:56:50.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.973+0000 7f31bbb18700 5 monclient: authenticate success, global_id 4484 2023-09-30T01:56:50.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.973+0000 7f31bbb18700 10 monclient: get_monmap_and_config success 2023-09-30T01:56:50.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.973+0000 7f31b88b2700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:50.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.973+0000 7f31b88b2700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:56:50.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.973+0000 7f31bbb18700 4 set_mon_vals no callback set 2023-09-30T01:56:50.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.973+0000 7f31b88b2700 10 monclient: dump: 2023-09-30T01:56:50.984 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:50.984 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:50.984 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:50.984 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:50.985 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:50.985 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:50.985 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:50.985 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:50.985 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:50.985 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:50.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.973+0000 7f31bbb18700 10 monclient: shutdown 2023-09-30T01:56:50.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.973+0000 7f31bbb18700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:50.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.973+0000 7f31bbb18700 1 -- 172.21.15.7:0/980349051 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f31b4097160 msgr2=0x7f31b4137550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:50.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.973+0000 7f31bbb18700 1 --2- 172.21.15.7:0/980349051 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f31b4097160 0x7f31b4137550 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f31a8009910 tx=0x7f31a80091a0 comp rx=0 tx=0).stop 2023-09-30T01:56:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.974+0000 7f31bbb18700 1 -- 172.21.15.7:0/980349051 shutdown_connections 2023-09-30T01:56:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.974+0000 7f31bbb18700 1 --2- 172.21.15.7:0/980349051 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f31b4097160 0x7f31b4137550 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.974+0000 7f31bbb18700 1 -- 172.21.15.7:0/980349051 >> 172.21.15.7:0/980349051 conn(0x7f31b40710d0 msgr2=0x7f31b40714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.974+0000 7f31bbb18700 1 -- 172.21.15.7:0/980349051 shutdown_connections 2023-09-30T01:56:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.974+0000 7f31bbb18700 1 -- 172.21.15.7:0/980349051 wait complete. 2023-09-30T01:56:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.974+0000 7f31bbb18700 10 monclient: build_initial_monmap 2023-09-30T01:56:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.974+0000 7f31bbb18700 10 monclient: monmap: 2023-09-30T01:56:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:56:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:56:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:56:50.986 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:56:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:56:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:56:50.987 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.974+0000 7f31bbb18700 1 Processor -- start 2023-09-30T01:56:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.974+0000 7f31bbb18700 1 -- start start 2023-09-30T01:56:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.974+0000 7f31bbb18700 10 monclient: init 2023-09-30T01:56:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31bbb18700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31bbb18700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:56:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31bbb18700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f31b4097160 0x7f31b411e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:50.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31bbb18700 10 monclient(hunting): picked mon.noname-c con 0x7f31b4097160 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:50.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31bbb18700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31b4097530 0x7f31b411e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:50.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31bbb18700 10 monclient(hunting): picked mon.noname-a con 0x7f31b4097530 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:56:50.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31bbb18700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31b40a29e0 0x7f31b411eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:50.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31bbb18700 10 monclient(hunting): picked mon.noname-b con 0x7f31b40a29e0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:56:50.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31bbb18700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:50.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31bbb18700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b4121e20 con 0x7f31b4097530 2023-09-30T01:56:50.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31bbb18700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:50.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31bbb18700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b40a1220 con 0x7f31b40a29e0 2023-09-30T01:56:50.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31bbb18700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:50.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31bbb18700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b40a0620 con 0x7f31b4097160 2023-09-30T01:56:50.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31bbb18700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:50.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31bbb18700 10 monclient(hunting): authenticate will time out at 1300.039429s 2023-09-30T01:56:50.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b90b3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31b4097530 0x7f31b411e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:50.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b90b3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31b4097530 0x7f31b411e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:45030/0 (socket says 172.21.15.7:45030) 2023-09-30T01:56:50.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b90b3700 1 -- 172.21.15.7:0/3793365489 learned_addr learned my addr 172.21.15.7:0/3793365489 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:50.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b90b3700 10 monclient(hunting): get_auth_request con 0x7f31b4097530 auth_method 0 2023-09-30T01:56:50.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b90b3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:50.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b90b3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:50.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b90b3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:50.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b98b4700 1 --2- 172.21.15.7:0/3793365489 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f31b4097160 0x7f31b411e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:50.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b98b4700 10 monclient(hunting): get_auth_request con 0x7f31b4097160 auth_method 0 2023-09-30T01:56:50.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b98b4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:50.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b98b4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:50.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b98b4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:50.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b90b3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:50.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b90b3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:50.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b90b3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:50.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b98b4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:50.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b98b4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:50.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b98b4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:50.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b98b4700 10 monclient(hunting): handle_auth_done global_id 4487 payload 824 2023-09-30T01:56:50.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31ba0b5700 1 --2- 172.21.15.7:0/3793365489 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31b40a29e0 0x7f31b411eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:50.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b98b4700 1 -- 172.21.15.7:0/3793365489 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31b40a29e0 msgr2=0x7f31b411eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:50.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b98b4700 1 --2- 172.21.15.7:0/3793365489 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31b40a29e0 0x7f31b411eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:50.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b98b4700 1 -- 172.21.15.7:0/3793365489 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31b4097530 msgr2=0x7f31b411e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:50.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b98b4700 1 --2- 172.21.15.7:0/3793365489 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31b4097530 0x7f31b411e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b98b4700 10 monclient: _finish_hunting 0 2023-09-30T01:56:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b98b4700 1 monclient: found mon.noname-c 2023-09-30T01:56:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b98b4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b98b4700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:56:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b98b4700 1 -- 172.21.15.7:0/3793365489 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a8009420 con 0x7f31b4097160 2023-09-30T01:56:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.975+0000 7f31b90b3700 1 --2- 172.21.15.7:0/3793365489 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31b4097530 0x7f31b411e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:56:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31b98b4700 1 -- 172.21.15.7:0/3793365489 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f31b4097160 msgr2=0x7f31b411e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31b98b4700 1 --2- 172.21.15.7:0/3793365489 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f31b4097160 0x7f31b411e430 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f31a400c8b0 tx=0x7f31a4002690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31b1ffb700 1 -- 172.21.15.7:0/3793365489 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a40070f0 con 0x7f31b4097160 2023-09-30T01:56:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31b1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:56:50.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31b1ffb700 10 monclient: dump: 2023-09-30T01:56:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:50.992 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31b1ffb700 10 monclient: _finish_auth 0 2023-09-30T01:56:50.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31b1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31b1ffb700 1 -- 172.21.15.7:0/3793365489 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a4007270 con 0x7f31b4097160 2023-09-30T01:56:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31bbb18700 5 monclient: authenticate success, global_id 4487 2023-09-30T01:56:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31bbb18700 10 monclient: _renew_subs 2023-09-30T01:56:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31bbb18700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31bbb18700 1 -- 172.21.15.7:0/3793365489 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b4120d50 con 0x7f31b4097160 2023-09-30T01:56:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31b1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31b1ffb700 1 -- 172.21.15.7:0/3793365489 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a4007630 con 0x7f31b4097160 2023-09-30T01:56:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31b1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:56:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31b1ffb700 10 monclient: dump: 2023-09-30T01:56:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:50.993 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:50.994 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31b3fff700 4 set_mon_vals no callback set 2023-09-30T01:56:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31bbb18700 10 monclient: _renew_subs 2023-09-30T01:56:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31bbb18700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:50.994 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.978+0000 7f31bbb18700 1 -- 172.21.15.7:0/3793365489 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b4130bc0 con 0x7f31b4097160 2023-09-30T01:56:50.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.979+0000 7f31b1ffb700 1 -- 172.21.15.7:0/3793365489 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f31a401b400 con 0x7f31b4097160 2023-09-30T01:56:50.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.979+0000 7f31b1ffb700 1 --2- 172.21.15.7:0/3793365489 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f31a00357e0 0x7f31a0037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:50.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.979+0000 7f31b1ffb700 1 -- 172.21.15.7:0/3793365489 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f31a40478f0 con 0x7f31b4097160 2023-09-30T01:56:50.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.979+0000 7f31bbb18700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:50.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.979+0000 7f31bbb18700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:50.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.979+0000 7f31bbb18700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:50.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.979+0000 7f31bbb18700 1 -- 172.21.15.7:0/3793365489 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b4120cf0 con 0x7f31b4097160 2023-09-30T01:56:50.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.979+0000 7f31b90b3700 1 --2- 172.21.15.7:0/3793365489 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f31a00357e0 0x7f31a0037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:50.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.980+0000 7f31b90b3700 10 monclient: get_auth_request con 0x7f31a00357e0 auth_method 0 2023-09-30T01:56:50.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.982+0000 7f31b90b3700 1 --2- 172.21.15.7:0/3793365489 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f31a00357e0 0x7f31a0037c90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f31b41204b0 tx=0x7f31a8016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:50.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.984+0000 7f31b1ffb700 1 -- 172.21.15.7:0/3793365489 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f31a401a360 con 0x7f31b4097160 2023-09-30T01:56:50.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.984+0000 7f31b1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:50.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:50.984+0000 7f31b1ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:56:51.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:51.220+0000 7f31bbb18700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:56:51.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:51.220+0000 7f31bbb18700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:51.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:51.220+0000 7f31bbb18700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:51.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:51.220+0000 7f31bbb18700 1 -- 172.21.15.7:0/3793365489 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f31b4064c10 con 0x7f31b4097160 2023-09-30T01:56:51.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:51.221+0000 7f31b1ffb700 1 -- 172.21.15.7:0/3793365489 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f31a400ae60 con 0x7f31b4097160 2023-09-30T01:56:51.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:51.221+0000 7f31b1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:51.220 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:51.221+0000 7f31b1ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:56:51.221 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:56:51.221 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:56:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:51.222+0000 7f31bbb18700 1 -- 172.21.15.7:0/3793365489 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f31a00357e0 msgr2=0x7f31a0037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:51.222+0000 7f31bbb18700 1 --2- 172.21.15.7:0/3793365489 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f31a00357e0 0x7f31a0037c90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f31b41204b0 tx=0x7f31a8016000 comp rx=0 tx=0).stop 2023-09-30T01:56:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:51.222+0000 7f31bbb18700 10 monclient: shutdown 2023-09-30T01:56:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:51.222+0000 7f31bbb18700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:51.222+0000 7f31bbb18700 1 -- 172.21.15.7:0/3793365489 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f31b4097160 msgr2=0x7f31b411e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:51.222+0000 7f31bbb18700 1 --2- 172.21.15.7:0/3793365489 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f31b4097160 0x7f31b411e430 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f31a400c8b0 tx=0x7f31a4002690 comp rx=0 tx=0).stop 2023-09-30T01:56:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:51.222+0000 7f31bbb18700 1 -- 172.21.15.7:0/3793365489 shutdown_connections 2023-09-30T01:56:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:51.222+0000 7f31bbb18700 1 --2- 172.21.15.7:0/3793365489 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f31a00357e0 0x7f31a0037c90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:51.222+0000 7f31bbb18700 1 --2- 172.21.15.7:0/3793365489 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31b40a29e0 0x7f31b411eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:51.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:51.222+0000 7f31bbb18700 1 --2- 172.21.15.7:0/3793365489 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f31b4097160 0x7f31b411e430 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:51.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:51.222+0000 7f31bbb18700 1 --2- 172.21.15.7:0/3793365489 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31b4097530 0x7f31b411e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:51.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:51.222+0000 7f31bbb18700 1 -- 172.21.15.7:0/3793365489 >> 172.21.15.7:0/3793365489 conn(0x7f31b40710d0 msgr2=0x7f31b4099ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:51.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:51.222+0000 7f31bbb18700 1 -- 172.21.15.7:0/3793365489 shutdown_connections 2023-09-30T01:56:51.224 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:51.222+0000 7f31bbb18700 1 -- 172.21.15.7:0/3793365489 wait complete. 2023-09-30T01:56:54.234 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:56:54.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.362+0000 7f5e764aa700 10 monclient: get_monmap_and_config 2023-09-30T01:56:54.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.362+0000 7f5e764aa700 10 monclient: build_initial_monmap 2023-09-30T01:56:54.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.362+0000 7f5e764aa700 10 monclient: monmap: 2023-09-30T01:56:54.363 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:54.363 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:54.363 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:56:54.363604+0000 2023-09-30T01:56:54.363 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:56:54.363604+0000 2023-09-30T01:56:54.363 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:54.363 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:54.363 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:56:54.363 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:56:54.363 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:56:54.363 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:56:54.364 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:56:54.364 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:54.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.363+0000 7f5e764aa700 1 Processor -- start 2023-09-30T01:56:54.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.363+0000 7f5e764aa700 1 -- start start 2023-09-30T01:56:54.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.363+0000 7f5e764aa700 10 monclient: init 2023-09-30T01:56:54.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.363+0000 7f5e764aa700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:54.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.363+0000 7f5e764aa700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-09-30T01:56:54.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.363+0000 7f5e764aa700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5e70096a50 0x7f5e70132750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:54.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.363+0000 7f5e764aa700 10 monclient(hunting): picked mon.noname-c con 0x7f5e70096a50 addr v2:172.21.15.50:3300/0 2023-09-30T01:56:54.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.363+0000 7f5e764aa700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e70096e20 addr v1:172.21.15.50:6789/0 2023-09-30T01:56:54.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.363+0000 7f5e764aa700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e701376f0 0x7f5e70139ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:54.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.363+0000 7f5e764aa700 10 monclient(hunting): picked mon.noname-b con 0x7f5e701376f0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:54.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.363+0000 7f5e764aa700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e70099480 con 0x7f5e70096e20 2023-09-30T01:56:54.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.363+0000 7f5e764aa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:54.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.363+0000 7f5e764aa700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e70097bf0 con 0x7f5e70096a50 2023-09-30T01:56:54.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.363+0000 7f5e764aa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:54.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.363+0000 7f5e764aa700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e700991a0 con 0x7f5e701376f0 2023-09-30T01:56:54.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.363+0000 7f5e764aa700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:54.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.363+0000 7f5e764aa700 10 monclient(hunting): authenticate will time out at 1303.428845s 2023-09-30T01:56:54.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.363+0000 7f5e74a47700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e701376f0 0x7f5e70139ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:54.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e74a47700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e701376f0 0x7f5e70139ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:39552/0 (socket says 172.21.15.7:39552) 2023-09-30T01:56:54.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e74a47700 1 -- 172.21.15.7:0/329178537 learned_addr learned my addr 172.21.15.7:0/329178537 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:54.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e74a47700 10 monclient(hunting): get_auth_request con 0x7f5e701376f0 auth_method 0 2023-09-30T01:56:54.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e74a47700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:54.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e74a47700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:54.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e74a47700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:54.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e6ffff700 1 --2- 172.21.15.7:0/329178537 >> v2:172.21.15.50:3300/0 conn(0x7f5e70096a50 0x7f5e70132750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:54.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e6ffff700 10 monclient(hunting): get_auth_request con 0x7f5e70096a50 auth_method 0 2023-09-30T01:56:54.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e6ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:54.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e6ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:54.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e6ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:54.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e74a47700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:54.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e74a47700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:54.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e74a47700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:54.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e6ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:54.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e6ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:54.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e6ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:54.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e74a47700 10 monclient(hunting): handle_auth_done global_id 4490 payload 274 2023-09-30T01:56:54.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e74a47700 1 -- 172.21.15.7:0/329178537 >> v2:172.21.15.50:3300/0 conn(0x7f5e70096a50 msgr2=0x7f5e70132750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:54.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e74a47700 1 --2- 172.21.15.7:0/329178537 >> v2:172.21.15.50:3300/0 conn(0x7f5e70096a50 0x7f5e70132750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:54.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e74a47700 1 -- 172.21.15.7:0/329178537 >> v1:172.21.15.50:6789/0 conn(0x7f5e70096e20 legacy=0x7f5e70097240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:54.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e74a47700 10 monclient: _finish_hunting 0 2023-09-30T01:56:54.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e74a47700 1 monclient: found mon.noname-b 2023-09-30T01:56:54.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e74a47700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:54.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e74a47700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:56:54.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.364+0000 7f5e74a47700 1 -- 172.21.15.7:0/329178537 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e70099760 con 0x7f5e701376f0 2023-09-30T01:56:54.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.366+0000 7f5e74a47700 1 -- 172.21.15.7:0/329178537 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e701376f0 msgr2=0x7f5e70139ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:54.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.366+0000 7f5e74a47700 1 --2- 172.21.15.7:0/329178537 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e701376f0 0x7f5e70139ad0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009f20 tx=0x7f5e68009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8359cefdece3e00d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:54.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.366+0000 7f5e6effd700 1 -- 172.21.15.7:0/329178537 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6800e070 con 0x7f5e701376f0 2023-09-30T01:56:54.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.366+0000 7f5e6effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:54.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.366+0000 7f5e6effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:56:54.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.366+0000 7f5e6effd700 10 monclient: dump: 2023-09-30T01:56:54.375 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:54.375 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:54.375 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:54.375 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:54.375 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:54.375 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:54.375 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:54.375 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:54.375 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:54.376 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:54.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.366+0000 7f5e6effd700 10 monclient: _finish_auth 0 2023-09-30T01:56:54.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.366+0000 7f5e6effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:54.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.366+0000 7f5e764aa700 5 monclient: authenticate success, global_id 4490 2023-09-30T01:56:54.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.366+0000 7f5e6effd700 1 -- 172.21.15.7:0/329178537 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6800bba0 con 0x7f5e701376f0 2023-09-30T01:56:54.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.366+0000 7f5e6effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:54.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.366+0000 7f5e764aa700 10 monclient: get_monmap_and_config success 2023-09-30T01:56:54.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.366+0000 7f5e764aa700 4 set_mon_vals no callback set 2023-09-30T01:56:54.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.366+0000 7f5e6effd700 1 -- 172.21.15.7:0/329178537 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e680092e0 con 0x7f5e701376f0 2023-09-30T01:56:54.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.366+0000 7f5e6effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:54.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.366+0000 7f5e6effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:56:54.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.366+0000 7f5e6effd700 10 monclient: dump: 2023-09-30T01:56:54.377 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:54.377 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:54.377 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:54.377 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:54.377 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:54.377 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:54.377 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:54.377 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:54.377 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:54.377 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:54.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.367+0000 7f5e764aa700 10 monclient: shutdown 2023-09-30T01:56:54.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.367+0000 7f5e764aa700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:54.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.367+0000 7f5e764aa700 1 -- 172.21.15.7:0/329178537 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e701376f0 msgr2=0x7f5e70139ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:54.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.367+0000 7f5e764aa700 1 --2- 172.21.15.7:0/329178537 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e701376f0 0x7f5e70139ad0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009f20 tx=0x7f5e68009840 comp rx=0 tx=0).stop 2023-09-30T01:56:54.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.367+0000 7f5e764aa700 1 -- 172.21.15.7:0/329178537 shutdown_connections 2023-09-30T01:56:54.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.367+0000 7f5e764aa700 1 --2- 172.21.15.7:0/329178537 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e701376f0 0x7f5e70139ad0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:54.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.367+0000 7f5e764aa700 1 --2- 172.21.15.7:0/329178537 >> v2:172.21.15.50:3300/0 conn(0x7f5e70096a50 0x7f5e70132750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:54.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.367+0000 7f5e764aa700 1 -- 172.21.15.7:0/329178537 >> 172.21.15.7:0/329178537 conn(0x7f5e70070cf0 msgr2=0x7f5e700710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:54.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.367+0000 7f5e764aa700 1 -- 172.21.15.7:0/329178537 shutdown_connections 2023-09-30T01:56:54.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.367+0000 7f5e764aa700 1 -- 172.21.15.7:0/329178537 wait complete. 2023-09-30T01:56:54.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.367+0000 7f5e764aa700 10 monclient: build_initial_monmap 2023-09-30T01:56:54.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.367+0000 7f5e764aa700 10 monclient: monmap: 2023-09-30T01:56:54.378 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:54.379 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:56:54.379 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:56:54.379 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:56:54.379 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:54.379 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:54.379 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:56:54.379 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:56:54.379 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:56:54.379 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:54.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.368+0000 7f5e764aa700 1 Processor -- start 2023-09-30T01:56:54.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.368+0000 7f5e764aa700 1 -- start start 2023-09-30T01:56:54.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.368+0000 7f5e764aa700 10 monclient: init 2023-09-30T01:56:54.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.368+0000 7f5e764aa700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:54.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.368+0000 7f5e764aa700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:56:54.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.368+0000 7f5e764aa700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e70096a50 0x7f5e700a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:54.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.368+0000 7f5e764aa700 10 monclient(hunting): picked mon.noname-c con 0x7f5e70096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:54.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.368+0000 7f5e764aa700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e70096e20 0x7f5e7009ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:54.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.368+0000 7f5e764aa700 10 monclient(hunting): picked mon.noname-b con 0x7f5e70096e20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:56:54.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.368+0000 7f5e764aa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e701376f0 0x7f5e7009cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:54.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e764aa700 10 monclient(hunting): picked mon.noname-a con 0x7f5e701376f0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:56:54.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e764aa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:54.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e764aa700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7009def0 con 0x7f5e701376f0 2023-09-30T01:56:54.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e764aa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:54.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e764aa700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e700a0770 con 0x7f5e70096e20 2023-09-30T01:56:54.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e764aa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:54.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e764aa700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e701334c0 con 0x7f5e70096a50 2023-09-30T01:56:54.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e764aa700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:54.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e764aa700 10 monclient(hunting): authenticate will time out at 1303.434035s 2023-09-30T01:56:54.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6ffff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e70096a50 0x7f5e700a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:54.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6ffff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e70096a50 0x7f5e700a1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:39568/0 (socket says 172.21.15.7:39568) 2023-09-30T01:56:54.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6ffff700 1 -- 172.21.15.7:0/2822068840 learned_addr learned my addr 172.21.15.7:0/2822068840 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:54.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6ffff700 10 monclient(hunting): get_auth_request con 0x7f5e70096a50 auth_method 0 2023-09-30T01:56:54.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:54.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:54.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:54.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e74a47700 1 --2- 172.21.15.7:0/2822068840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e701376f0 0x7f5e7009cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:54.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e74a47700 10 monclient(hunting): get_auth_request con 0x7f5e701376f0 auth_method 0 2023-09-30T01:56:54.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e74a47700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:54.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e74a47700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:54.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e74a47700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:54.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:54.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:54.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:54.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e74a47700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:54.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e74a47700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:54.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e74a47700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:54.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6f7fe700 1 --2- 172.21.15.7:0/2822068840 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e70096e20 0x7f5e7009ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:54.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6f7fe700 10 monclient(hunting): get_auth_request con 0x7f5e70096e20 auth_method 0 2023-09-30T01:56:54.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:54.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:54.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:54.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6ffff700 10 monclient(hunting): handle_auth_done global_id 4493 payload 824 2023-09-30T01:56:54.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6ffff700 1 -- 172.21.15.7:0/2822068840 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e70096e20 msgr2=0x7f5e7009ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:54.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6ffff700 1 --2- 172.21.15.7:0/2822068840 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e70096e20 0x7f5e7009ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:54.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6ffff700 1 -- 172.21.15.7:0/2822068840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e701376f0 msgr2=0x7f5e7009cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:54.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6ffff700 1 --2- 172.21.15.7:0/2822068840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e701376f0 0x7f5e7009cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:54.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6ffff700 10 monclient: _finish_hunting 0 2023-09-30T01:56:54.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6ffff700 1 monclient: found mon.noname-c 2023-09-30T01:56:54.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:54.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:56:54.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6ffff700 1 -- 172.21.15.7:0/2822068840 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e68009ae0 con 0x7f5e70096a50 2023-09-30T01:56:54.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e6f7fe700 1 --2- 172.21.15.7:0/2822068840 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e70096e20 0x7f5e7009ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:56:54.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.369+0000 7f5e74a47700 1 --2- 172.21.15.7:0/2822068840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e701376f0 0x7f5e7009cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:56:54.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e6ffff700 1 -- 172.21.15.7:0/2822068840 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e70096a50 msgr2=0x7f5e700a1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:54.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e6ffff700 1 --2- 172.21.15.7:0/2822068840 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e70096a50 0x7f5e700a1130 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5e6400ec80 tx=0x7f5e640091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:54.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e57fff700 1 -- 172.21.15.7:0/2822068840 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6400ca60 con 0x7f5e70096a50 2023-09-30T01:56:54.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e57fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:54.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e57fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:56:54.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e57fff700 10 monclient: dump: 2023-09-30T01:56:54.387 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:54.387 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:54.387 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:54.387 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:54.387 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:54.387 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:54.387 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:54.388 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:54.388 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:54.388 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:54.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e57fff700 10 monclient: _finish_auth 0 2023-09-30T01:56:54.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e57fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:54.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e764aa700 5 monclient: authenticate success, global_id 4493 2023-09-30T01:56:54.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e57fff700 1 -- 172.21.15.7:0/2822068840 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6400cbc0 con 0x7f5e70096a50 2023-09-30T01:56:54.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e57fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:54.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e57fff700 1 -- 172.21.15.7:0/2822068840 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e64013470 con 0x7f5e70096a50 2023-09-30T01:56:54.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e57fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:54.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e57fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:56:54.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e57fff700 10 monclient: dump: 2023-09-30T01:56:54.389 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:54.389 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:54.389 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:54.389 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:54.389 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:54.389 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:54.389 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:54.389 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:54.389 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:54.390 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:54.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e764aa700 10 monclient: _renew_subs 2023-09-30T01:56:54.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e764aa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:54.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e764aa700 1 -- 172.21.15.7:0/2822068840 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e70131e20 con 0x7f5e70096a50 2023-09-30T01:56:54.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e6e7fc700 4 set_mon_vals no callback set 2023-09-30T01:56:54.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e764aa700 10 monclient: _renew_subs 2023-09-30T01:56:54.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e764aa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:54.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.371+0000 7f5e764aa700 1 -- 172.21.15.7:0/2822068840 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e7009ebd0 con 0x7f5e70096a50 2023-09-30T01:56:54.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.372+0000 7f5e57fff700 1 -- 172.21.15.7:0/2822068840 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f5e640137e0 con 0x7f5e70096a50 2023-09-30T01:56:54.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.372+0000 7f5e57fff700 1 --2- 172.21.15.7:0/2822068840 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5e5c0357e0 0x7f5e5c037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:54.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.372+0000 7f5e57fff700 1 -- 172.21.15.7:0/2822068840 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f5e64047e30 con 0x7f5e70096a50 2023-09-30T01:56:54.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.372+0000 7f5e764aa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:54.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.372+0000 7f5e764aa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:54.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.372+0000 7f5e764aa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:54.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.372+0000 7f5e764aa700 1 -- 172.21.15.7:0/2822068840 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e700a05d0 con 0x7f5e70096a50 2023-09-30T01:56:54.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.373+0000 7f5e6f7fe700 1 --2- 172.21.15.7:0/2822068840 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5e5c0357e0 0x7f5e5c037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:54.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.373+0000 7f5e6f7fe700 10 monclient: get_auth_request con 0x7f5e5c0357e0 auth_method 0 2023-09-30T01:56:54.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.375+0000 7f5e6f7fe700 1 --2- 172.21.15.7:0/2822068840 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5e5c0357e0 0x7f5e5c037c90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5e60007fd0 tx=0x7f5e60009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:54.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.377+0000 7f5e57fff700 1 -- 172.21.15.7:0/2822068840 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5e700a05d0 con 0x7f5e70096a50 2023-09-30T01:56:54.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.377+0000 7f5e57fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:54.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.377+0000 7f5e57fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:56:54.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.614+0000 7f5e764aa700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:56:54.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.614+0000 7f5e764aa700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:54.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.614+0000 7f5e764aa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:54.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.614+0000 7f5e764aa700 1 -- 172.21.15.7:0/2822068840 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5e700a05d0 con 0x7f5e70096a50 2023-09-30T01:56:54.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.614+0000 7f5e57fff700 1 -- 172.21.15.7:0/2822068840 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5e700a05d0 con 0x7f5e70096a50 2023-09-30T01:56:54.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.614+0000 7f5e57fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:54.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.614+0000 7f5e57fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:56:54.615 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:56:54.615 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:56:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.616+0000 7f5e764aa700 1 -- 172.21.15.7:0/2822068840 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5e5c0357e0 msgr2=0x7f5e5c037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.616+0000 7f5e764aa700 1 --2- 172.21.15.7:0/2822068840 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5e5c0357e0 0x7f5e5c037c90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5e60007fd0 tx=0x7f5e60009040 comp rx=0 tx=0).stop 2023-09-30T01:56:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.616+0000 7f5e764aa700 10 monclient: shutdown 2023-09-30T01:56:54.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.616+0000 7f5e764aa700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:54.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.616+0000 7f5e764aa700 1 -- 172.21.15.7:0/2822068840 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e70096a50 msgr2=0x7f5e700a1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:54.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.616+0000 7f5e764aa700 1 --2- 172.21.15.7:0/2822068840 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e70096a50 0x7f5e700a1130 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5e6400ec80 tx=0x7f5e640091a0 comp rx=0 tx=0).stop 2023-09-30T01:56:54.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.616+0000 7f5e764aa700 1 -- 172.21.15.7:0/2822068840 shutdown_connections 2023-09-30T01:56:54.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.616+0000 7f5e764aa700 1 --2- 172.21.15.7:0/2822068840 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5e5c0357e0 0x7f5e5c037c90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:54.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.616+0000 7f5e764aa700 1 --2- 172.21.15.7:0/2822068840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5e701376f0 0x7f5e7009cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:54.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.616+0000 7f5e764aa700 1 --2- 172.21.15.7:0/2822068840 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5e70096a50 0x7f5e700a1130 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:54.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.616+0000 7f5e764aa700 1 --2- 172.21.15.7:0/2822068840 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e70096e20 0x7f5e7009ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:54.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.616+0000 7f5e764aa700 1 -- 172.21.15.7:0/2822068840 >> 172.21.15.7:0/2822068840 conn(0x7f5e70070cf0 msgr2=0x7f5e700a4500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:54.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.616+0000 7f5e764aa700 1 -- 172.21.15.7:0/2822068840 shutdown_connections 2023-09-30T01:56:54.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:54.616+0000 7f5e764aa700 1 -- 172.21.15.7:0/2822068840 wait complete. 2023-09-30T01:56:57.628 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:56:57.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.758+0000 7f6f5f59e700 10 monclient: get_monmap_and_config 2023-09-30T01:56:57.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.758+0000 7f6f5f59e700 10 monclient: build_initial_monmap 2023-09-30T01:56:57.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.758+0000 7f6f5f59e700 10 monclient: monmap: 2023-09-30T01:56:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:56:57.759910+0000 2023-09-30T01:56:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:56:57.759910+0000 2023-09-30T01:56:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:56:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:56:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:56:57.760 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:56:57.761 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:56:57.761 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:57.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.759+0000 7f6f5f59e700 1 Processor -- start 2023-09-30T01:56:57.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.759+0000 7f6f5f59e700 1 -- start start 2023-09-30T01:56:57.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.759+0000 7f6f5f59e700 10 monclient: init 2023-09-30T01:56:57.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5f59e700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:57.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5f59e700 10 monclient: _add_conns ranks=[3,0,1,4,2] 2023-09-30T01:56:57.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5f59e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6f60098540 0x7f6f601329e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:57.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f6f60098540 addr v2:172.21.15.7:3300/0 2023-09-30T01:56:57.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5f59e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6f60098910 addr v1:172.21.15.7:6789/0 2023-09-30T01:56:57.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5f59e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6f60098d10 addr v1:172.21.15.50:6789/0 2023-09-30T01:56:57.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5f59e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f60069f10 con 0x7f6f60098910 2023-09-30T01:56:57.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5f59e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f60096a50 con 0x7f6f60098d10 2023-09-30T01:56:57.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5f59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:57.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5f59e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f600a0f40 con 0x7f6f60098540 2023-09-30T01:56:57.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5f59e700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:57.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5f59e700 10 monclient(hunting): authenticate will time out at 1306.825163s 2023-09-30T01:56:57.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5e59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6f60098540 0x7f6f601329e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:57.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5dd9b700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6f60098910 0x7f6f6012ebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58372/0 (socket says 172.21.15.7:58372) 2023-09-30T01:56:57.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5dd9b700 1 -- 172.21.15.7:0/93040157 learned_addr learned my addr 172.21.15.7:0/93040157 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:56:57.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5e59c700 10 monclient(hunting): get_auth_request con 0x7f6f60098540 auth_method 0 2023-09-30T01:56:57.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:57.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:57.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:57.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:57.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:57.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:57.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5e59c700 10 monclient(hunting): handle_auth_done global_id 4479 payload 274 2023-09-30T01:56:57.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5e59c700 1 -- 172.21.15.7:0/93040157 >> v1:172.21.15.50:6789/0 conn(0x7f6f60098d10 legacy=0x7f6f6012f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:57.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5e59c700 1 -- 172.21.15.7:0/93040157 >> v1:172.21.15.7:6789/0 conn(0x7f6f60098910 legacy=0x7f6f6012ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:56:57.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5e59c700 10 monclient: _finish_hunting 0 2023-09-30T01:56:57.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5e59c700 1 monclient: found mon.noname-a 2023-09-30T01:56:57.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:57.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5e59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:56:57.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.760+0000 7f6f5e59c700 1 -- 172.21.15.7:0/93040157 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f600a05c0 con 0x7f6f60098540 2023-09-30T01:56:57.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.762+0000 7f6f5e59c700 1 -- 172.21.15.7:0/93040157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f60098540 msgr2=0x7f6f601329e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:57.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.762+0000 7f6f5e59c700 1 --2- 172.21.15.7:0/93040157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f60098540 0x7f6f601329e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6f50009e60 tx=0x7f6f50009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3bd687dc13b21ec server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:57.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.762+0000 7f6f5d59a700 1 -- 172.21.15.7:0/93040157 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5000e070 con 0x7f6f60098540 2023-09-30T01:56:57.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.762+0000 7f6f5d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:57.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5d59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:56:57.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5d59a700 10 monclient: dump: 2023-09-30T01:56:57.775 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:57.775 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:57.775 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:57.775 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:57.775 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:57.775 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:57.775 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:57.775 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:57.775 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:57.776 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:57.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5d59a700 10 monclient: _finish_auth 0 2023-09-30T01:56:57.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:57.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5d59a700 1 -- 172.21.15.7:0/93040157 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f50012430 con 0x7f6f60098540 2023-09-30T01:56:57.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5f59e700 5 monclient: authenticate success, global_id 4479 2023-09-30T01:56:57.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:56:57.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:57.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5d59a700 1 -- 172.21.15.7:0/93040157 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5000bea0 con 0x7f6f60098540 2023-09-30T01:56:57.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5f59e700 10 monclient: get_monmap_and_config success 2023-09-30T01:56:57.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5f59e700 4 set_mon_vals no callback set 2023-09-30T01:56:57.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:57.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5d59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:56:57.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5d59a700 10 monclient: dump: 2023-09-30T01:56:57.777 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:57.777 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:57.777 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:57.777 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:57.777 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:57.777 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:57.778 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:57.778 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:57.778 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:57.778 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:57.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5f59e700 10 monclient: shutdown 2023-09-30T01:56:57.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5f59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:57.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5f59e700 1 -- 172.21.15.7:0/93040157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f60098540 msgr2=0x7f6f601329e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:57.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5f59e700 1 --2- 172.21.15.7:0/93040157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f60098540 0x7f6f601329e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6f50009e60 tx=0x7f6f50009780 comp rx=0 tx=0).stop 2023-09-30T01:56:57.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5f59e700 1 -- 172.21.15.7:0/93040157 shutdown_connections 2023-09-30T01:56:57.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5f59e700 1 --2- 172.21.15.7:0/93040157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f60098540 0x7f6f601329e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:57.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5f59e700 1 -- 172.21.15.7:0/93040157 >> 172.21.15.7:0/93040157 conn(0x7f6f60071830 msgr2=0x7f6f600a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:57.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5f59e700 1 -- 172.21.15.7:0/93040157 shutdown_connections 2023-09-30T01:56:57.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.763+0000 7f6f5f59e700 1 -- 172.21.15.7:0/93040157 wait complete. 2023-09-30T01:56:57.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.764+0000 7f6f5f59e700 10 monclient: build_initial_monmap 2023-09-30T01:56:57.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.764+0000 7f6f5f59e700 10 monclient: monmap: 2023-09-30T01:56:57.780 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:56:57.780 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:56:57.780 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:56:57.780 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:56:57.780 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:56:57.780 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:57.780 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:56:57.781 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:56:57.781 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:56:57.781 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:57.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.764+0000 7f6f5f59e700 1 Processor -- start 2023-09-30T01:56:57.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.764+0000 7f6f5f59e700 1 -- start start 2023-09-30T01:56:57.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.764+0000 7f6f5f59e700 10 monclient: init 2023-09-30T01:56:57.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5f59e700 10 monclient: _reopen_session rank -1 2023-09-30T01:56:57.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5f59e700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:56:57.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5f59e700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f60098540 0x7f6f6009ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:57.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f6f60098540 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:56:57.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5f59e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f60098910 0x7f6f6011ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:57.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f6f60098910 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:56:57.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5f59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f60098d10 0x7f6f6011bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:57.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f6f60098d10 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:56:57.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5f59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:57.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5f59e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f601327b0 con 0x7f6f60098910 2023-09-30T01:56:57.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5f59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:57.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5f59e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6011cac0 con 0x7f6f60098d10 2023-09-30T01:56:57.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5f59e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:56:57.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5f59e700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6012da00 con 0x7f6f60098540 2023-09-30T01:56:57.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5f59e700 10 monclient(hunting): _renew_subs 2023-09-30T01:56:57.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5f59e700 10 monclient(hunting): authenticate will time out at 1306.830184s 2023-09-30T01:56:57.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5e59c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f60098540 0x7f6f6009ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:57.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5dd9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f60098910 0x7f6f6011ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:57.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5e59c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f60098540 0x7f6f6009ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:39590/0 (socket says 172.21.15.7:39590) 2023-09-30T01:56:57.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5e59c700 1 -- 172.21.15.7:0/2331334286 learned_addr learned my addr 172.21.15.7:0/2331334286 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:56:57.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5e59c700 10 monclient(hunting): get_auth_request con 0x7f6f60098540 auth_method 0 2023-09-30T01:56:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5dd9b700 1 --2- 172.21.15.7:0/2331334286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f60098910 0x7f6f6011ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59210/0 (socket says 172.21.15.7:59210) 2023-09-30T01:56:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5dd9b700 10 monclient(hunting): get_auth_request con 0x7f6f60098910 auth_method 0 2023-09-30T01:56:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:57.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:57.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:56:57.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:56:57.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:56:57.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5ed9d700 1 --2- 172.21.15.7:0/2331334286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f60098d10 0x7f6f6011bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:57.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5ed9d700 10 monclient(hunting): get_auth_request con 0x7f6f60098d10 auth_method 0 2023-09-30T01:56:57.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:56:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:56:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:56:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5e59c700 10 monclient(hunting): handle_auth_done global_id 4496 payload 824 2023-09-30T01:56:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5e59c700 1 -- 172.21.15.7:0/2331334286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f60098d10 msgr2=0x7f6f6011bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5e59c700 1 --2- 172.21.15.7:0/2331334286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f60098d10 0x7f6f6011bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5e59c700 1 -- 172.21.15.7:0/2331334286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f60098910 msgr2=0x7f6f6011ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5e59c700 1 --2- 172.21.15.7:0/2331334286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f60098910 0x7f6f6011ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5e59c700 10 monclient: _finish_hunting 0 2023-09-30T01:56:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5e59c700 1 monclient: found mon.noname-c 2023-09-30T01:56:57.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:56:57.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5e59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:56:57.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.765+0000 7f6f5e59c700 1 -- 172.21.15.7:0/2331334286 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f50009a20 con 0x7f6f60098540 2023-09-30T01:56:57.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.767+0000 7f6f5e59c700 1 -- 172.21.15.7:0/2331334286 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f60098540 msgr2=0x7f6f6009ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:56:57.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.767+0000 7f6f5e59c700 1 --2- 172.21.15.7:0/2331334286 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f60098540 0x7f6f6009ec00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6f6006b6b0 tx=0x7f6f50018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:57.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.767+0000 7f6f4e7fc700 1 -- 172.21.15.7:0/2331334286 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f500070e0 con 0x7f6f60098540 2023-09-30T01:56:57.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.767+0000 7f6f4e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:57.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.767+0000 7f6f4e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:56:57.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.767+0000 7f6f4e7fc700 10 monclient: dump: 2023-09-30T01:56:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:57.789 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:57.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.767+0000 7f6f4e7fc700 10 monclient: _finish_auth 0 2023-09-30T01:56:57.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.768+0000 7f6f4e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:56:57.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.768+0000 7f6f5f59e700 5 monclient: authenticate success, global_id 4496 2023-09-30T01:56:57.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.768+0000 7f6f4e7fc700 1 -- 172.21.15.7:0/2331334286 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f50021070 con 0x7f6f60098540 2023-09-30T01:56:57.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.768+0000 7f6f4e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:56:57.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.768+0000 7f6f4e7fc700 1 -- 172.21.15.7:0/2331334286 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f50026430 con 0x7f6f60098540 2023-09-30T01:56:57.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.768+0000 7f6f4e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:56:57.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.768+0000 7f6f4e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:56:57.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.768+0000 7f6f4e7fc700 10 monclient: dump: 2023-09-30T01:56:57.790 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:56:57.790 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:56:57.790 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:57.790 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:56:57.790 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:56:57.790 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:56:57.790 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:56:57.790 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:56:57.791 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:56:57.791 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:56:57.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.768+0000 7f6f5cd99700 4 set_mon_vals no callback set 2023-09-30T01:56:57.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.768+0000 7f6f5f59e700 10 monclient: _renew_subs 2023-09-30T01:56:57.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.768+0000 7f6f5f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:57.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.768+0000 7f6f5f59e700 1 -- 172.21.15.7:0/2331334286 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f6011d340 con 0x7f6f60098540 2023-09-30T01:56:57.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.768+0000 7f6f5f59e700 10 monclient: _renew_subs 2023-09-30T01:56:57.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.768+0000 7f6f5f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:57.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.768+0000 7f6f5f59e700 1 -- 172.21.15.7:0/2331334286 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f6012ea10 con 0x7f6f60098540 2023-09-30T01:56:57.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.768+0000 7f6f4e7fc700 1 -- 172.21.15.7:0/2331334286 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f6f50023030 con 0x7f6f60098540 2023-09-30T01:56:57.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.768+0000 7f6f5f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:57.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.768+0000 7f6f5f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:57.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.768+0000 7f6f5f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:57.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.768+0000 7f6f5f59e700 1 -- 172.21.15.7:0/2331334286 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f6011eb90 con 0x7f6f60098540 2023-09-30T01:56:57.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.768+0000 7f6f4e7fc700 1 --2- 172.21.15.7:0/2331334286 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6f40035810 0x7f6f40037cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:56:57.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.769+0000 7f6f4e7fc700 1 -- 172.21.15.7:0/2331334286 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f6f50049f70 con 0x7f6f60098540 2023-09-30T01:56:57.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.769+0000 7f6f5dd9b700 1 --2- 172.21.15.7:0/2331334286 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6f40035810 0x7f6f40037cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:56:57.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.769+0000 7f6f5dd9b700 10 monclient: get_auth_request con 0x7f6f40035810 auth_method 0 2023-09-30T01:56:57.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.771+0000 7f6f5dd9b700 1 --2- 172.21.15.7:0/2331334286 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6f40035810 0x7f6f40037cc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6f6009f420 tx=0x7f6f54007f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:56:57.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.773+0000 7f6f4e7fc700 1 -- 172.21.15.7:0/2331334286 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6f5001c420 con 0x7f6f60098540 2023-09-30T01:56:57.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.773+0000 7f6f4e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:56:57.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:57.773+0000 7f6f4e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:56:58.007 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:58.007+0000 7f6f5f59e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:56:58.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:58.007+0000 7f6f5f59e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:58.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:58.007+0000 7f6f5f59e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:56:58.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:58.007+0000 7f6f5f59e700 1 -- 172.21.15.7:0/2331334286 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6f6011eb90 con 0x7f6f60098540 2023-09-30T01:56:58.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:58.007+0000 7f6f4e7fc700 1 -- 172.21.15.7:0/2331334286 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6f5001b3a0 con 0x7f6f60098540 2023-09-30T01:56:58.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:58.007+0000 7f6f4e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:56:58.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:58.007+0000 7f6f4e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:56:58.008 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:56:58.009 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:56:58.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:58.010+0000 7f6f5f59e700 1 -- 172.21.15.7:0/2331334286 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6f40035810 msgr2=0x7f6f40037cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:58.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:58.010+0000 7f6f5f59e700 1 --2- 172.21.15.7:0/2331334286 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6f40035810 0x7f6f40037cc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6f6009f420 tx=0x7f6f54007f90 comp rx=0 tx=0).stop 2023-09-30T01:56:58.010 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:58.010+0000 7f6f5f59e700 10 monclient: shutdown 2023-09-30T01:56:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:58.010+0000 7f6f5f59e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:56:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:58.010+0000 7f6f5f59e700 1 -- 172.21.15.7:0/2331334286 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f60098540 msgr2=0x7f6f6009ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:56:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:58.010+0000 7f6f5f59e700 1 --2- 172.21.15.7:0/2331334286 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f60098540 0x7f6f6009ec00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6f6006b6b0 tx=0x7f6f50018000 comp rx=0 tx=0).stop 2023-09-30T01:56:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:58.010+0000 7f6f5f59e700 1 -- 172.21.15.7:0/2331334286 shutdown_connections 2023-09-30T01:56:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:58.010+0000 7f6f5f59e700 1 --2- 172.21.15.7:0/2331334286 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6f40035810 0x7f6f40037cc0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:58.010+0000 7f6f5f59e700 1 --2- 172.21.15.7:0/2331334286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6f60098d10 0x7f6f6011bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:58.010+0000 7f6f5f59e700 1 --2- 172.21.15.7:0/2331334286 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6f60098540 0x7f6f6009ec00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:58.010+0000 7f6f5f59e700 1 --2- 172.21.15.7:0/2331334286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6f60098910 0x7f6f6011ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:56:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:58.010+0000 7f6f5f59e700 1 -- 172.21.15.7:0/2331334286 >> 172.21.15.7:0/2331334286 conn(0x7f6f60071830 msgr2=0x7f6f60131910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:56:58.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:58.010+0000 7f6f5f59e700 1 -- 172.21.15.7:0/2331334286 shutdown_connections 2023-09-30T01:56:58.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:56:58.010+0000 7f6f5f59e700 1 -- 172.21.15.7:0/2331334286 wait complete. 2023-09-30T01:57:01.023 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:57:01.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.153+0000 7f6d32b3e700 10 monclient: get_monmap_and_config 2023-09-30T01:57:01.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.153+0000 7f6d32b3e700 10 monclient: build_initial_monmap 2023-09-30T01:57:01.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.153+0000 7f6d32b3e700 10 monclient: monmap: 2023-09-30T01:57:01.153 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:01.153 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:01.153 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:57:01.154060+0000 2023-09-30T01:57:01.153 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:57:01.154060+0000 2023-09-30T01:57:01.153 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:01.153 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:01.153 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:57:01.154 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:57:01.154 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:57:01.154 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:57:01.154 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:57:01.154 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:01.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.153+0000 7f6d32b3e700 1 Processor -- start 2023-09-30T01:57:01.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.153+0000 7f6d32b3e700 1 -- start start 2023-09-30T01:57:01.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.153+0000 7f6d32b3e700 10 monclient: init 2023-09-30T01:57:01.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d32b3e700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:01.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d32b3e700 10 monclient: _add_conns ranks=[3,2,0,4,1] 2023-09-30T01:57:01.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d32b3e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6d2c096a50 0x7f6d2c1327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:01.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d32b3e700 10 monclient(hunting): picked mon.noname-a con 0x7f6d2c096a50 addr v2:172.21.15.7:3300/0 2023-09-30T01:57:01.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d32b3e700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d2c0a2390 0x7f6d2c096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:01.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d32b3e700 10 monclient(hunting): picked mon.noname-b con 0x7f6d2c0a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:01.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d32b3e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d2c097360 addr v1:172.21.15.7:6789/0 2023-09-30T01:57:01.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d32b3e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d2c0980e0 con 0x7f6d2c097360 2023-09-30T01:57:01.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d32b3e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:01.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d32b3e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d2c0991a0 con 0x7f6d2c096a50 2023-09-30T01:57:01.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d32b3e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:01.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d32b3e700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c099480 con 0x7f6d2c0a2390 2023-09-30T01:57:01.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d32b3e700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:01.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d32b3e700 10 monclient(hunting): authenticate will time out at 1310.219262s 2023-09-30T01:57:01.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d308da700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6d2c096a50 0x7f6d2c1327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:01.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d310db700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f6d2c097360 0x7f6d2c12ef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58386/0 (socket says 172.21.15.7:58386) 2023-09-30T01:57:01.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d2bfff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d2c0a2390 0x7f6d2c096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:01.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d310db700 1 -- 172.21.15.7:0/2368396576 learned_addr learned my addr 172.21.15.7:0/2368396576 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:57:01.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d308da700 10 monclient(hunting): get_auth_request con 0x7f6d2c096a50 auth_method 0 2023-09-30T01:57:01.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d308da700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:01.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d308da700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:01.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d308da700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:01.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d2bfff700 10 monclient(hunting): get_auth_request con 0x7f6d2c0a2390 auth_method 0 2023-09-30T01:57:01.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d2bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:01.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d2bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:01.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d2bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:01.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d308da700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:01.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d308da700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:01.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d308da700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:01.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d2bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:01.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d2bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:01.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d2bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:01.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d2bfff700 10 monclient(hunting): handle_auth_done global_id 4499 payload 274 2023-09-30T01:57:01.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d2bfff700 1 -- 172.21.15.7:0/2368396576 >> v2:172.21.15.7:3300/0 conn(0x7f6d2c096a50 msgr2=0x7f6d2c1327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:01.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d2bfff700 1 --2- 172.21.15.7:0/2368396576 >> v2:172.21.15.7:3300/0 conn(0x7f6d2c096a50 0x7f6d2c1327a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:01.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d2bfff700 1 -- 172.21.15.7:0/2368396576 >> v1:172.21.15.7:6789/0 conn(0x7f6d2c097360 legacy=0x7f6d2c12ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:01.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d2bfff700 10 monclient: _finish_hunting 0 2023-09-30T01:57:01.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d2bfff700 1 monclient: found mon.noname-b 2023-09-30T01:57:01.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d2bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:01.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d2bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:57:01.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d2bfff700 1 -- 172.21.15.7:0/2368396576 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d2c097bf0 con 0x7f6d2c0a2390 2023-09-30T01:57:01.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.154+0000 7f6d308da700 1 --2- 172.21.15.7:0/2368396576 >> v2:172.21.15.7:3300/0 conn(0x7f6d2c096a50 0x7f6d2c1327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:01.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d2bfff700 1 -- 172.21.15.7:0/2368396576 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d2c0a2390 msgr2=0x7f6d2c096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:01.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d2bfff700 1 --2- 172.21.15.7:0/2368396576 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d2c0a2390 0x7f6d2c096e20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6d180098e0 tx=0x7f6d180091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dba65b481ecfee41 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:01.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d2b7fe700 1 -- 172.21.15.7:0/2368396576 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d18011840 con 0x7f6d2c0a2390 2023-09-30T01:57:01.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d2b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:01.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d2b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:57:01.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d2b7fe700 10 monclient: dump: 2023-09-30T01:57:01.166 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:01.166 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:01.166 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:01.166 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:01.166 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:01.166 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:01.166 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:01.166 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:01.166 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:01.166 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:01.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d2b7fe700 10 monclient: _finish_auth 0 2023-09-30T01:57:01.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d2b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:01.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d2b7fe700 1 -- 172.21.15.7:0/2368396576 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d180119a0 con 0x7f6d2c0a2390 2023-09-30T01:57:01.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d32b3e700 5 monclient: authenticate success, global_id 4499 2023-09-30T01:57:01.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d32b3e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:57:01.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d2b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:01.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d2b7fe700 1 -- 172.21.15.7:0/2368396576 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d18011d40 con 0x7f6d2c0a2390 2023-09-30T01:57:01.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d2b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:01.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d2b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:57:01.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d2b7fe700 10 monclient: dump: 2023-09-30T01:57:01.167 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:01.167 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:01.168 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:01.168 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:01.168 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:01.168 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:01.168 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:01.168 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:01.168 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:01.168 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:01.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d32b3e700 10 monclient: get_monmap_and_config success 2023-09-30T01:57:01.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d32b3e700 4 set_mon_vals no callback set 2023-09-30T01:57:01.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d32b3e700 10 monclient: shutdown 2023-09-30T01:57:01.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d32b3e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:01.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d32b3e700 1 -- 172.21.15.7:0/2368396576 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d2c0a2390 msgr2=0x7f6d2c096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:01.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d32b3e700 1 --2- 172.21.15.7:0/2368396576 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d2c0a2390 0x7f6d2c096e20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6d180098e0 tx=0x7f6d180091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:01.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d32b3e700 1 -- 172.21.15.7:0/2368396576 shutdown_connections 2023-09-30T01:57:01.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d32b3e700 1 --2- 172.21.15.7:0/2368396576 >> v2:172.21.15.7:3300/0 conn(0x7f6d2c096a50 0x7f6d2c1327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:01.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d32b3e700 1 --2- 172.21.15.7:0/2368396576 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d2c0a2390 0x7f6d2c096e20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:01.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d32b3e700 1 -- 172.21.15.7:0/2368396576 >> 172.21.15.7:0/2368396576 conn(0x7f6d2c070cf0 msgr2=0x7f6d2c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:01.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d32b3e700 1 -- 172.21.15.7:0/2368396576 shutdown_connections 2023-09-30T01:57:01.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.157+0000 7f6d32b3e700 1 -- 172.21.15.7:0/2368396576 wait complete. 2023-09-30T01:57:01.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.158+0000 7f6d32b3e700 10 monclient: build_initial_monmap 2023-09-30T01:57:01.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.158+0000 7f6d32b3e700 10 monclient: monmap: 2023-09-30T01:57:01.178 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:01.178 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:57:01.178 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:57:01.178 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:57:01.178 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:01.179 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:01.179 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:57:01.179 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:57:01.179 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:57:01.179 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:01.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.158+0000 7f6d32b3e700 1 Processor -- start 2023-09-30T01:57:01.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.158+0000 7f6d32b3e700 1 -- start start 2023-09-30T01:57:01.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.158+0000 7f6d32b3e700 10 monclient: init 2023-09-30T01:57:01.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d32b3e700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:01.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d32b3e700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:57:01.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d32b3e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d2c096a50 0x7f6d2c12def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:01.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d32b3e700 10 monclient(hunting): picked mon.noname-a con 0x7f6d2c096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:57:01.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d32b3e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d2c097360 0x7f6d2c09db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:01.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d32b3e700 10 monclient(hunting): picked mon.noname-b con 0x7f6d2c097360 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:57:01.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d32b3e700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d2c0a2390 0x7f6d2c09e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:01.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d32b3e700 10 monclient(hunting): picked mon.noname-c con 0x7f6d2c0a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:01.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d32b3e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:01.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d32b3e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c1334e0 con 0x7f6d2c096a50 2023-09-30T01:57:01.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d32b3e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:01.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d32b3e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c12f0a0 con 0x7f6d2c097360 2023-09-30T01:57:01.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d32b3e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:01.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d32b3e700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c09fc40 con 0x7f6d2c0a2390 2023-09-30T01:57:01.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d32b3e700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:01.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d32b3e700 10 monclient(hunting): authenticate will time out at 1310.224395s 2023-09-30T01:57:01.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d308da700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d2c096a50 0x7f6d2c12def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:01.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d308da700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d2c096a50 0x7f6d2c12def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59230/0 (socket says 172.21.15.7:59230) 2023-09-30T01:57:01.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d308da700 1 -- 172.21.15.7:0/930331495 learned_addr learned my addr 172.21.15.7:0/930331495 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:01.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d308da700 10 monclient(hunting): get_auth_request con 0x7f6d2c096a50 auth_method 0 2023-09-30T01:57:01.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d308da700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:01.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d308da700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:01.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d308da700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:01.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d310db700 1 --2- 172.21.15.7:0/930331495 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d2c0a2390 0x7f6d2c09e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:01.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d310db700 10 monclient(hunting): get_auth_request con 0x7f6d2c0a2390 auth_method 0 2023-09-30T01:57:01.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d310db700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:01.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d310db700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:01.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d310db700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d308da700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d308da700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d308da700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d310db700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d310db700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d2bfff700 1 --2- 172.21.15.7:0/930331495 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d2c097360 0x7f6d2c09db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d310db700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d2bfff700 10 monclient(hunting): get_auth_request con 0x7f6d2c097360 auth_method 0 2023-09-30T01:57:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d2bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d2bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d2bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:01.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d2bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:01.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d2bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:01.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d2bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:01.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d308da700 10 monclient(hunting): handle_auth_done global_id 4488 payload 824 2023-09-30T01:57:01.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d308da700 1 -- 172.21.15.7:0/930331495 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d2c0a2390 msgr2=0x7f6d2c09e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:01.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d308da700 1 --2- 172.21.15.7:0/930331495 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d2c0a2390 0x7f6d2c09e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:01.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d308da700 1 -- 172.21.15.7:0/930331495 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d2c097360 msgr2=0x7f6d2c09db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:01.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d308da700 1 --2- 172.21.15.7:0/930331495 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d2c097360 0x7f6d2c09db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:01.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d308da700 10 monclient: _finish_hunting 0 2023-09-30T01:57:01.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d308da700 1 monclient: found mon.noname-a 2023-09-30T01:57:01.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d308da700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:01.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d308da700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:57:01.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d308da700 1 -- 172.21.15.7:0/930331495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d20009420 con 0x7f6d2c096a50 2023-09-30T01:57:01.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.159+0000 7f6d310db700 1 --2- 172.21.15.7:0/930331495 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d2c0a2390 0x7f6d2c09e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:01.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d308da700 1 -- 172.21.15.7:0/930331495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d2c096a50 msgr2=0x7f6d2c12def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:01.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d308da700 1 --2- 172.21.15.7:0/930331495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d2c096a50 0x7f6d2c12def0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6d2000ec80 tx=0x7f6d200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:01.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d28ff9700 1 -- 172.21.15.7:0/930331495 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2000ca60 con 0x7f6d2c096a50 2023-09-30T01:57:01.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d28ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:01.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d28ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:57:01.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d28ff9700 10 monclient: dump: 2023-09-30T01:57:01.184 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:01.184 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:01.184 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:01.185 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:01.185 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:01.185 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:01.185 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:01.185 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:01.185 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:01.185 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:01.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d28ff9700 10 monclient: _finish_auth 0 2023-09-30T01:57:01.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d28ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:01.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d32b3e700 5 monclient: authenticate success, global_id 4488 2023-09-30T01:57:01.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d28ff9700 1 -- 172.21.15.7:0/930331495 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d2000cbc0 con 0x7f6d2c096a50 2023-09-30T01:57:01.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d32b3e700 10 monclient: _renew_subs 2023-09-30T01:57:01.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d32b3e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:01.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d32b3e700 1 -- 172.21.15.7:0/930331495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d18009420 con 0x7f6d2c096a50 2023-09-30T01:57:01.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d28ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:01.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d28ff9700 1 -- 172.21.15.7:0/930331495 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d20013470 con 0x7f6d2c096a50 2023-09-30T01:57:01.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d28ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:01.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d28ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:57:01.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d28ff9700 10 monclient: dump: 2023-09-30T01:57:01.186 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:01.186 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:01.186 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:01.186 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:01.186 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:01.187 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:01.187 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:01.187 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:01.187 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:01.187 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:01.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d32b3e700 10 monclient: _renew_subs 2023-09-30T01:57:01.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d2affd700 4 set_mon_vals no callback set 2023-09-30T01:57:01.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d32b3e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:01.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.162+0000 7f6d32b3e700 1 -- 172.21.15.7:0/930331495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d2c0a11f0 con 0x7f6d2c096a50 2023-09-30T01:57:01.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.163+0000 7f6d308da700 1 --2- 172.21.15.7:0/930331495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d2c096a50 0x7f6d2c12def0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6d2000ec80 tx=0x7f6d200091a0 comp rx=0 tx=0).queue_received will delay after 1.69604e+09 on 0x7f6d200480a0 osd_map(15..15 src has 1..15) v4 2023-09-30T01:57:01.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.163+0000 7f6d28ff9700 1 -- 172.21.15.7:0/930331495 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f6d200137e0 con 0x7f6d2c096a50 2023-09-30T01:57:01.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.163+0000 7f6d28ff9700 1 --2- 172.21.15.7:0/930331495 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6d10035830 0x7f6d10037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:01.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.163+0000 7f6d32b3e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:01.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.163+0000 7f6d32b3e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:01.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.163+0000 7f6d32b3e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:01.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.163+0000 7f6d32b3e700 1 -- 172.21.15.7:0/930331495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d2c0a4880 con 0x7f6d2c096a50 2023-09-30T01:57:01.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.163+0000 7f6d2bfff700 1 --2- 172.21.15.7:0/930331495 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6d10035830 0x7f6d10037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:01.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.163+0000 7f6d2bfff700 10 monclient: get_auth_request con 0x7f6d10035830 auth_method 0 2023-09-30T01:57:01.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.166+0000 7f6d2bfff700 1 --2- 172.21.15.7:0/930331495 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6d10035830 0x7f6d10037ce0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c06a090 tx=0x7f6d18016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:01.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.168+0000 7f6d28ff9700 1 -- 172.21.15.7:0/930331495 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f6d200480a0 con 0x7f6d2c096a50 2023-09-30T01:57:01.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.991+0000 7f6d28ff9700 1 -- 172.21.15.7:0/930331495 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6d200236b0 con 0x7f6d2c096a50 2023-09-30T01:57:01.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.991+0000 7f6d28ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:01.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:01.991+0000 7f6d28ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:57:02.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:02.234+0000 7f6d32b3e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:57:02.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:02.234+0000 7f6d32b3e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:02.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:02.234+0000 7f6d32b3e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:02.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:02.234+0000 7f6d32b3e700 1 -- 172.21.15.7:0/930331495 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6d2c0a1520 con 0x7f6d2c096a50 2023-09-30T01:57:02.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:02.235+0000 7f6d28ff9700 1 -- 172.21.15.7:0/930331495 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6d200232f0 con 0x7f6d2c096a50 2023-09-30T01:57:02.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:02.235+0000 7f6d28ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:02.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:02.235+0000 7f6d28ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:57:02.236 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:57:02.236 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:57:02.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:02.237+0000 7f6d32b3e700 1 -- 172.21.15.7:0/930331495 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6d10035830 msgr2=0x7f6d10037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:02.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:02.237+0000 7f6d32b3e700 1 --2- 172.21.15.7:0/930331495 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6d10035830 0x7f6d10037ce0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c06a090 tx=0x7f6d18016000 comp rx=0 tx=0).stop 2023-09-30T01:57:02.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:02.237+0000 7f6d32b3e700 10 monclient: shutdown 2023-09-30T01:57:02.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:02.237+0000 7f6d32b3e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:02.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:02.237+0000 7f6d32b3e700 1 -- 172.21.15.7:0/930331495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d2c096a50 msgr2=0x7f6d2c12def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:02.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:02.237+0000 7f6d32b3e700 1 --2- 172.21.15.7:0/930331495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d2c096a50 0x7f6d2c12def0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6d2000ec80 tx=0x7f6d200091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:02.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:02.237+0000 7f6d32b3e700 1 -- 172.21.15.7:0/930331495 shutdown_connections 2023-09-30T01:57:02.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:02.237+0000 7f6d32b3e700 1 --2- 172.21.15.7:0/930331495 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6d10035830 0x7f6d10037ce0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:02.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:02.237+0000 7f6d32b3e700 1 --2- 172.21.15.7:0/930331495 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d2c096a50 0x7f6d2c12def0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:02.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:02.237+0000 7f6d32b3e700 1 --2- 172.21.15.7:0/930331495 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d2c0a2390 0x7f6d2c09e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:02.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:02.237+0000 7f6d32b3e700 1 --2- 172.21.15.7:0/930331495 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d2c097360 0x7f6d2c09db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:02.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:02.237+0000 7f6d32b3e700 1 -- 172.21.15.7:0/930331495 >> 172.21.15.7:0/930331495 conn(0x7f6d2c070cf0 msgr2=0x7f6d2c131a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:02.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:02.237+0000 7f6d32b3e700 1 -- 172.21.15.7:0/930331495 shutdown_connections 2023-09-30T01:57:02.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:02.237+0000 7f6d32b3e700 1 -- 172.21.15.7:0/930331495 wait complete. 2023-09-30T01:57:05.249 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:57:05.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.379+0000 7fc6baa31700 10 monclient: get_monmap_and_config 2023-09-30T01:57:05.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.379+0000 7fc6baa31700 10 monclient: build_initial_monmap 2023-09-30T01:57:05.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.379+0000 7fc6baa31700 10 monclient: monmap: 2023-09-30T01:57:05.380 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:05.380 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:05.380 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:57:05.380938+0000 2023-09-30T01:57:05.380 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:57:05.380938+0000 2023-09-30T01:57:05.380 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:05.380 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:05.381 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:57:05.381 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:57:05.381 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:57:05.381 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:57:05.381 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:57:05.381 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:05.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.380+0000 7fc6baa31700 1 Processor -- start 2023-09-30T01:57:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.380+0000 7fc6baa31700 1 -- start start 2023-09-30T01:57:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.380+0000 7fc6baa31700 10 monclient: init 2023-09-30T01:57:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6baa31700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6baa31700 10 monclient: _add_conns ranks=[3,0,4,1,2] 2023-09-30T01:57:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6baa31700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc6b4098540 0x7fc6b41329e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6baa31700 10 monclient(hunting): picked mon.noname-a con 0x7fc6b4098540 addr v2:172.21.15.7:3300/0 2023-09-30T01:57:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6baa31700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc6b4098910 addr v1:172.21.15.7:6789/0 2023-09-30T01:57:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6baa31700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc6b4098d10 0x7fc6b412f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6baa31700 10 monclient(hunting): picked mon.noname-c con 0x7fc6b4098d10 addr v2:172.21.15.50:3300/0 2023-09-30T01:57:05.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6baa31700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b4069f10 con 0x7fc6b4098910 2023-09-30T01:57:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6baa31700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6baa31700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6b40a0f40 con 0x7fc6b4098540 2023-09-30T01:57:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6baa31700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6baa31700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6b4132f20 con 0x7fc6b4098d10 2023-09-30T01:57:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6baa31700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6baa31700 10 monclient(hunting): authenticate will time out at 1314.446332s 2023-09-30T01:57:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6b9a2f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc6b4098540 0x7fc6b41329e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6b922e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc6b4098910 0x7fc6b412ebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50004/0 (socket says 172.21.15.7:50004) 2023-09-30T01:57:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6b922e700 1 -- 172.21.15.7:0/1520463167 learned_addr learned my addr 172.21.15.7:0/1520463167 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:57:05.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6b9a2f700 10 monclient(hunting): get_auth_request con 0x7fc6b4098540 auth_method 0 2023-09-30T01:57:05.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6b9a2f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:05.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6b9a2f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:05.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6b9a2f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:05.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6b9a2f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:05.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6b9a2f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:05.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6b9a2f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:05.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6ba230700 1 --2- 172.21.15.7:0/1520463167 >> v2:172.21.15.50:3300/0 conn(0x7fc6b4098d10 0x7fc6b412f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:05.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6ba230700 10 monclient(hunting): get_auth_request con 0x7fc6b4098d10 auth_method 0 2023-09-30T01:57:05.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6ba230700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:05.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6ba230700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:05.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6ba230700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:05.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6b9a2f700 10 monclient(hunting): handle_auth_done global_id 4491 payload 274 2023-09-30T01:57:05.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6b9a2f700 1 -- 172.21.15.7:0/1520463167 >> v2:172.21.15.50:3300/0 conn(0x7fc6b4098d10 msgr2=0x7fc6b412f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:05.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6b9a2f700 1 --2- 172.21.15.7:0/1520463167 >> v2:172.21.15.50:3300/0 conn(0x7fc6b4098d10 0x7fc6b412f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:05.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6b9a2f700 1 -- 172.21.15.7:0/1520463167 >> v1:172.21.15.7:6789/0 conn(0x7fc6b4098910 legacy=0x7fc6b412ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:05.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6b9a2f700 10 monclient: _finish_hunting 0 2023-09-30T01:57:05.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6b9a2f700 1 monclient: found mon.noname-a 2023-09-30T01:57:05.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6b9a2f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:05.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6b9a2f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:57:05.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6b9a2f700 1 -- 172.21.15.7:0/1520463167 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b4096a50 con 0x7fc6b4098540 2023-09-30T01:57:05.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.381+0000 7fc6ba230700 1 --2- 172.21.15.7:0/1520463167 >> v2:172.21.15.50:3300/0 conn(0x7fc6b4098d10 0x7fc6b412f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:57:05.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6b9a2f700 1 -- 172.21.15.7:0/1520463167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6b4098540 msgr2=0x7fc6b41329e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:05.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6b9a2f700 1 --2- 172.21.15.7:0/1520463167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6b4098540 0x7fc6b41329e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc6b4097380 tx=0x7fc6b0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d779c9a99b7aa38 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:05.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6b8a2d700 1 -- 172.21.15.7:0/1520463167 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b000e070 con 0x7fc6b4098540 2023-09-30T01:57:05.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6b8a2d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:05.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6b8a2d700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:57:05.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6b8a2d700 10 monclient: dump: 2023-09-30T01:57:05.392 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:05.393 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:05.393 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:05.393 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:05.393 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:05.393 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:05.393 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:05.393 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:05.393 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:05.393 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:05.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6b8a2d700 10 monclient: _finish_auth 0 2023-09-30T01:57:05.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6b8a2d700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:05.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6b8a2d700 1 -- 172.21.15.7:0/1520463167 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b000be00 con 0x7fc6b4098540 2023-09-30T01:57:05.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6baa31700 5 monclient: authenticate success, global_id 4491 2023-09-30T01:57:05.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6baa31700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:57:05.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6b8a2d700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:05.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6b8a2d700 1 -- 172.21.15.7:0/1520463167 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b0012660 con 0x7fc6b4098540 2023-09-30T01:57:05.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6baa31700 10 monclient: get_monmap_and_config success 2023-09-30T01:57:05.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6baa31700 4 set_mon_vals no callback set 2023-09-30T01:57:05.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6b8a2d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:05.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6b8a2d700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:57:05.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6b8a2d700 10 monclient: dump: 2023-09-30T01:57:05.395 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:05.395 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:05.395 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:05.395 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:05.395 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:05.395 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:05.396 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:05.396 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:05.396 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:05.396 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:05.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6baa31700 10 monclient: shutdown 2023-09-30T01:57:05.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6baa31700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:05.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6baa31700 1 -- 172.21.15.7:0/1520463167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6b4098540 msgr2=0x7fc6b41329e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:05.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6baa31700 1 --2- 172.21.15.7:0/1520463167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6b4098540 0x7fc6b41329e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc6b4097380 tx=0x7fc6b0009780 comp rx=0 tx=0).stop 2023-09-30T01:57:05.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6baa31700 1 -- 172.21.15.7:0/1520463167 shutdown_connections 2023-09-30T01:57:05.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6baa31700 1 --2- 172.21.15.7:0/1520463167 >> v2:172.21.15.50:3300/0 conn(0x7fc6b4098d10 0x7fc6b412f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:05.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6baa31700 1 --2- 172.21.15.7:0/1520463167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6b4098540 0x7fc6b41329e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:05.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6baa31700 1 -- 172.21.15.7:0/1520463167 >> 172.21.15.7:0/1520463167 conn(0x7fc6b4071830 msgr2=0x7fc6b40a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:05.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6baa31700 1 -- 172.21.15.7:0/1520463167 shutdown_connections 2023-09-30T01:57:05.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.384+0000 7fc6baa31700 1 -- 172.21.15.7:0/1520463167 wait complete. 2023-09-30T01:57:05.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.385+0000 7fc6baa31700 10 monclient: build_initial_monmap 2023-09-30T01:57:05.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.385+0000 7fc6baa31700 10 monclient: monmap: 2023-09-30T01:57:05.397 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:05.397 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:57:05.397 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:57:05.398 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:57:05.398 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:05.398 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:05.398 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:57:05.398 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:57:05.398 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:57:05.398 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:05.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.385+0000 7fc6baa31700 1 Processor -- start 2023-09-30T01:57:05.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.385+0000 7fc6baa31700 1 -- start start 2023-09-30T01:57:05.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.385+0000 7fc6baa31700 10 monclient: init 2023-09-30T01:57:05.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6baa31700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:05.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6baa31700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:57:05.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6baa31700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc6b4098540 0x7fc6b411c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:05.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6baa31700 10 monclient(hunting): picked mon.noname-c con 0x7fc6b4098540 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:05.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6baa31700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b4098910 0x7fc6b411c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:05.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6baa31700 10 monclient(hunting): picked mon.noname-b con 0x7fc6b4098910 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:57:05.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6baa31700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6b4098d10 0x7fc6b411cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:05.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6baa31700 10 monclient(hunting): picked mon.noname-a con 0x7fc6b4098d10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:57:05.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6baa31700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:05.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6baa31700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b41327b0 con 0x7fc6b4098d10 2023-09-30T01:57:05.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6baa31700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:05.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6baa31700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b40a4010 con 0x7fc6b4098910 2023-09-30T01:57:05.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6baa31700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:05.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6baa31700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b411fb00 con 0x7fc6b4098540 2023-09-30T01:57:05.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6baa31700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:05.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6baa31700 10 monclient(hunting): authenticate will time out at 1314.451210s 2023-09-30T01:57:05.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6b9a2f700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc6b4098540 0x7fc6b411c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:05.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6ba230700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6b4098d10 0x7fc6b411cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:05.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6b9a2f700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc6b4098540 0x7fc6b411c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:51788/0 (socket says 172.21.15.7:51788) 2023-09-30T01:57:05.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6ba230700 10 monclient(hunting): get_auth_request con 0x7fc6b4098d10 auth_method 0 2023-09-30T01:57:05.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6b9a2f700 1 -- 172.21.15.7:0/1354337180 learned_addr learned my addr 172.21.15.7:0/1354337180 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:05.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6ba230700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:05.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6ba230700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:05.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6ba230700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:05.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6b9a2f700 10 monclient(hunting): get_auth_request con 0x7fc6b4098540 auth_method 0 2023-09-30T01:57:05.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6b9a2f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:05.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6b9a2f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:05.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6b9a2f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:05.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6ba230700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:05.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6ba230700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:05.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6ba230700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:05.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6b9a2f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:05.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6b9a2f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:05.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6b9a2f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:05.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6b922e700 1 --2- 172.21.15.7:0/1354337180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b4098910 0x7fc6b411c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:05.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6ba230700 10 monclient(hunting): handle_auth_done global_id 4494 payload 824 2023-09-30T01:57:05.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6ba230700 1 -- 172.21.15.7:0/1354337180 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc6b4098540 msgr2=0x7fc6b411c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:05.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6ba230700 1 --2- 172.21.15.7:0/1354337180 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc6b4098540 0x7fc6b411c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:05.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6ba230700 1 -- 172.21.15.7:0/1354337180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b4098910 msgr2=0x7fc6b411c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:05.402 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6ba230700 1 --2- 172.21.15.7:0/1354337180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b4098910 0x7fc6b411c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:05.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6ba230700 10 monclient: _finish_hunting 0 2023-09-30T01:57:05.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6ba230700 1 monclient: found mon.noname-a 2023-09-30T01:57:05.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6ba230700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:05.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6ba230700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:57:05.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6ba230700 1 -- 172.21.15.7:0/1354337180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b0009a20 con 0x7fc6b4098d10 2023-09-30T01:57:05.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6b922e700 10 monclient: get_auth_request con 0x7fc6b4098910 auth_method 0 2023-09-30T01:57:05.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.386+0000 7fc6b922e700 1 --2- 172.21.15.7:0/1354337180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b4098910 0x7fc6b411c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:57:05.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.388+0000 7fc6ba230700 1 -- 172.21.15.7:0/1354337180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6b4098d10 msgr2=0x7fc6b411cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:05.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.388+0000 7fc6ba230700 1 --2- 172.21.15.7:0/1354337180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6b4098d10 0x7fc6b411cd00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac00e870 tx=0x7fc6ac008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:05.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.389+0000 7fc6a9ffb700 1 -- 172.21.15.7:0/1354337180 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac013620 con 0x7fc6b4098d10 2023-09-30T01:57:05.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.389+0000 7fc6a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:05.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.389+0000 7fc6a9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:57:05.406 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.389+0000 7fc6a9ffb700 10 monclient: dump: 2023-09-30T01:57:05.406 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:05.406 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:05.406 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:05.407 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:05.407 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:05.407 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:05.407 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:05.407 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:05.407 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:05.407 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:05.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.389+0000 7fc6a9ffb700 10 monclient: _finish_auth 0 2023-09-30T01:57:05.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.389+0000 7fc6a9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:05.407 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.389+0000 7fc6baa31700 5 monclient: authenticate success, global_id 4494 2023-09-30T01:57:05.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.389+0000 7fc6a9ffb700 1 -- 172.21.15.7:0/1354337180 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6ac013780 con 0x7fc6b4098d10 2023-09-30T01:57:05.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.389+0000 7fc6a9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:05.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.389+0000 7fc6a9ffb700 1 -- 172.21.15.7:0/1354337180 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac013b20 con 0x7fc6b4098d10 2023-09-30T01:57:05.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.389+0000 7fc6baa31700 10 monclient: _renew_subs 2023-09-30T01:57:05.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.389+0000 7fc6baa31700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:05.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.389+0000 7fc6baa31700 1 -- 172.21.15.7:0/1354337180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b40a05c0 con 0x7fc6b4098d10 2023-09-30T01:57:05.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.389+0000 7fc6abfff700 4 set_mon_vals no callback set 2023-09-30T01:57:05.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.389+0000 7fc6a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:05.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.389+0000 7fc6a9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:57:05.408 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.389+0000 7fc6a9ffb700 10 monclient: dump: 2023-09-30T01:57:05.409 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:05.409 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:05.409 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:05.409 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:05.409 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:05.409 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:05.409 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:05.409 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:05.409 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:05.409 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:05.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.389+0000 7fc6baa31700 10 monclient: _renew_subs 2023-09-30T01:57:05.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.389+0000 7fc6baa31700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:05.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.389+0000 7fc6baa31700 1 -- 172.21.15.7:0/1354337180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b411dc60 con 0x7fc6b4098d10 2023-09-30T01:57:05.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.389+0000 7fc6a9ffb700 1 -- 172.21.15.7:0/1354337180 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fc6ac012370 con 0x7fc6b4098d10 2023-09-30T01:57:05.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.390+0000 7fc6a9ffb700 1 --2- 172.21.15.7:0/1354337180 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc698035810 0x7fc698037cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:05.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.390+0000 7fc6a9ffb700 1 -- 172.21.15.7:0/1354337180 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fc6ac047a00 con 0x7fc6b4098d10 2023-09-30T01:57:05.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.390+0000 7fc6baa31700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:05.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.390+0000 7fc6baa31700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:05.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.390+0000 7fc6baa31700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:05.410 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.390+0000 7fc6baa31700 1 -- 172.21.15.7:0/1354337180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b411fe90 con 0x7fc6b4098d10 2023-09-30T01:57:05.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.390+0000 7fc6b9a2f700 1 --2- 172.21.15.7:0/1354337180 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc698035810 0x7fc698037cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:05.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.390+0000 7fc6b9a2f700 10 monclient: get_auth_request con 0x7fc698035810 auth_method 0 2023-09-30T01:57:05.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.393+0000 7fc6b9a2f700 1 --2- 172.21.15.7:0/1354337180 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc698035810 0x7fc698037cc0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0009870 tx=0x7fc6b0018000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:05.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.395+0000 7fc6a9ffb700 1 -- 172.21.15.7:0/1354337180 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc6b411fe90 con 0x7fc6b4098d10 2023-09-30T01:57:05.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.395+0000 7fc6a9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:05.411 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.395+0000 7fc6a9ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:57:05.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.628+0000 7fc6baa31700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:57:05.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.628+0000 7fc6baa31700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:05.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.628+0000 7fc6baa31700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:05.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.628+0000 7fc6baa31700 1 -- 172.21.15.7:0/1354337180 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc6b411fe90 con 0x7fc6b4098d10 2023-09-30T01:57:05.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.628+0000 7fc6a9ffb700 1 -- 172.21.15.7:0/1354337180 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc6b411fe90 con 0x7fc6b4098d10 2023-09-30T01:57:05.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.628+0000 7fc6a9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:05.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.628+0000 7fc6a9ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:57:05.629 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:57:05.629 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:57:05.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.630+0000 7fc6baa31700 1 -- 172.21.15.7:0/1354337180 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc698035810 msgr2=0x7fc698037cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:05.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.630+0000 7fc6baa31700 1 --2- 172.21.15.7:0/1354337180 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc698035810 0x7fc698037cc0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0009870 tx=0x7fc6b0018000 comp rx=0 tx=0).stop 2023-09-30T01:57:05.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.630+0000 7fc6baa31700 10 monclient: shutdown 2023-09-30T01:57:05.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.630+0000 7fc6baa31700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:05.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.630+0000 7fc6baa31700 1 -- 172.21.15.7:0/1354337180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6b4098d10 msgr2=0x7fc6b411cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:05.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.630+0000 7fc6baa31700 1 --2- 172.21.15.7:0/1354337180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6b4098d10 0x7fc6b411cd00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac00e870 tx=0x7fc6ac008040 comp rx=0 tx=0).stop 2023-09-30T01:57:05.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.630+0000 7fc6baa31700 1 -- 172.21.15.7:0/1354337180 shutdown_connections 2023-09-30T01:57:05.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.630+0000 7fc6baa31700 1 --2- 172.21.15.7:0/1354337180 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc698035810 0x7fc698037cc0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:05.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.630+0000 7fc6baa31700 1 --2- 172.21.15.7:0/1354337180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc6b4098d10 0x7fc6b411cd00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:05.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.630+0000 7fc6baa31700 1 --2- 172.21.15.7:0/1354337180 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc6b4098540 0x7fc6b411c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:05.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.630+0000 7fc6baa31700 1 --2- 172.21.15.7:0/1354337180 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc6b4098910 0x7fc6b411c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:05.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.630+0000 7fc6baa31700 1 -- 172.21.15.7:0/1354337180 >> 172.21.15.7:0/1354337180 conn(0x7fc6b4071830 msgr2=0x7fc6b4131800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:05.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.630+0000 7fc6baa31700 1 -- 172.21.15.7:0/1354337180 shutdown_connections 2023-09-30T01:57:05.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:05.630+0000 7fc6baa31700 1 -- 172.21.15.7:0/1354337180 wait complete. 2023-09-30T01:57:08.642 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:57:08.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.773+0000 7f7ffe0f6700 10 monclient: get_monmap_and_config 2023-09-30T01:57:08.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.773+0000 7f7ffe0f6700 10 monclient: build_initial_monmap 2023-09-30T01:57:08.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.773+0000 7f7ffe0f6700 10 monclient: monmap: 2023-09-30T01:57:08.774 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:08.774 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:08.774 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:57:08.774997+0000 2023-09-30T01:57:08.774 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:57:08.774997+0000 2023-09-30T01:57:08.774 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:08.774 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:08.775 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:57:08.775 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:57:08.775 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:57:08.775 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:57:08.775 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:57:08.775 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:08.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.774+0000 7f7ffe0f6700 1 Processor -- start 2023-09-30T01:57:08.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.774+0000 7f7ffe0f6700 1 -- start start 2023-09-30T01:57:08.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.774+0000 7f7ffe0f6700 10 monclient: init 2023-09-30T01:57:08.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ffe0f6700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:08.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ffe0f6700 10 monclient: _add_conns ranks=[4,2,0,3,1] 2023-09-30T01:57:08.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ffe0f6700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7ff8096a50 0x7f7ff81327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:08.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ffe0f6700 10 monclient(hunting): picked mon.noname-c con 0x7f7ff8096a50 addr v2:172.21.15.50:3300/0 2023-09-30T01:57:08.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ffe0f6700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7ff80a2390 0x7f7ff8096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:08.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ffe0f6700 10 monclient(hunting): picked mon.noname-b con 0x7f7ff80a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:08.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ffe0f6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7ff8097360 addr v1:172.21.15.7:6789/0 2023-09-30T01:57:08.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ffe0f6700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff80980e0 con 0x7f7ff8097360 2023-09-30T01:57:08.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ffe0f6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:08.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ffe0f6700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ff80991a0 con 0x7f7ff8096a50 2023-09-30T01:57:08.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ffe0f6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:08.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ffe0f6700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff8099480 con 0x7f7ff80a2390 2023-09-30T01:57:08.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ffe0f6700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:08.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ffe0f6700 10 monclient(hunting): authenticate will time out at 1317.840385s 2023-09-30T01:57:08.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff7fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7ff8097360 0x7f7ff812ef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50020/0 (socket says 172.21.15.7:50020) 2023-09-30T01:57:08.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff7fff700 1 -- 172.21.15.7:0/4190262396 learned_addr learned my addr 172.21.15.7:0/4190262396 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:57:08.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff6ffd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7ff80a2390 0x7f7ff8096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:08.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff6ffd700 10 monclient(hunting): get_auth_request con 0x7f7ff80a2390 auth_method 0 2023-09-30T01:57:08.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:08.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:08.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:08.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:08.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:08.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff77fe700 1 --2- 172.21.15.7:0/4190262396 >> v2:172.21.15.50:3300/0 conn(0x7f7ff8096a50 0x7f7ff81327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:08.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:08.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff77fe700 10 monclient(hunting): get_auth_request con 0x7f7ff8096a50 auth_method 0 2023-09-30T01:57:08.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:08.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:08.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:08.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff6ffd700 10 monclient(hunting): handle_auth_done global_id 4508 payload 274 2023-09-30T01:57:08.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff6ffd700 1 -- 172.21.15.7:0/4190262396 >> v2:172.21.15.50:3300/0 conn(0x7f7ff8096a50 msgr2=0x7f7ff81327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:08.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff6ffd700 1 --2- 172.21.15.7:0/4190262396 >> v2:172.21.15.50:3300/0 conn(0x7f7ff8096a50 0x7f7ff81327a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:08.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff6ffd700 1 -- 172.21.15.7:0/4190262396 >> v1:172.21.15.7:6789/0 conn(0x7f7ff8097360 legacy=0x7f7ff812ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:08.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff6ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:57:08.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff6ffd700 1 monclient: found mon.noname-b 2023-09-30T01:57:08.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:08.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff6ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:57:08.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff6ffd700 1 -- 172.21.15.7:0/4190262396 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff8097bf0 con 0x7f7ff80a2390 2023-09-30T01:57:08.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.775+0000 7f7ff77fe700 1 --2- 172.21.15.7:0/4190262396 >> v2:172.21.15.50:3300/0 conn(0x7f7ff8096a50 0x7f7ff81327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:57:08.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ff6ffd700 1 -- 172.21.15.7:0/4190262396 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7ff80a2390 msgr2=0x7f7ff8096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:08.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ff6ffd700 1 --2- 172.21.15.7:0/4190262396 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7ff80a2390 0x7f7ff8096e20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0098e0 tx=0x7f7fec0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce461793fe56f0db server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:08.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ff67fc700 1 -- 172.21.15.7:0/4190262396 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec009e80 con 0x7f7ff80a2390 2023-09-30T01:57:08.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ff67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:08.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ff67fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:57:08.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ff67fc700 10 monclient: dump: 2023-09-30T01:57:08.783 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:08.784 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ff67fc700 10 monclient: _finish_auth 0 2023-09-30T01:57:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ff67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:08.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ffe0f6700 5 monclient: authenticate success, global_id 4508 2023-09-30T01:57:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ffe0f6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:57:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ff67fc700 1 -- 172.21.15.7:0/4190262396 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fec00ba30 con 0x7f7ff80a2390 2023-09-30T01:57:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ff67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ffe0f6700 10 monclient: get_monmap_and_config success 2023-09-30T01:57:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ffe0f6700 4 set_mon_vals no callback set 2023-09-30T01:57:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ff67fc700 1 -- 172.21.15.7:0/4190262396 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec00bdb0 con 0x7f7ff80a2390 2023-09-30T01:57:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ff67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ff67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:57:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ff67fc700 10 monclient: dump: 2023-09-30T01:57:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:08.785 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:08.786 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ffe0f6700 10 monclient: shutdown 2023-09-30T01:57:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ffe0f6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ffe0f6700 1 -- 172.21.15.7:0/4190262396 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7ff80a2390 msgr2=0x7f7ff8096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:08.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ffe0f6700 1 --2- 172.21.15.7:0/4190262396 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7ff80a2390 0x7f7ff8096e20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0098e0 tx=0x7f7fec0091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ffe0f6700 1 -- 172.21.15.7:0/4190262396 shutdown_connections 2023-09-30T01:57:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ffe0f6700 1 --2- 172.21.15.7:0/4190262396 >> v2:172.21.15.50:3300/0 conn(0x7f7ff8096a50 0x7f7ff81327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ffe0f6700 1 --2- 172.21.15.7:0/4190262396 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7ff80a2390 0x7f7ff8096e20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ffe0f6700 1 -- 172.21.15.7:0/4190262396 >> 172.21.15.7:0/4190262396 conn(0x7f7ff8070cf0 msgr2=0x7f7ff80710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ffe0f6700 1 -- 172.21.15.7:0/4190262396 shutdown_connections 2023-09-30T01:57:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.778+0000 7f7ffe0f6700 1 -- 172.21.15.7:0/4190262396 wait complete. 2023-09-30T01:57:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.779+0000 7f7ffe0f6700 10 monclient: build_initial_monmap 2023-09-30T01:57:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.779+0000 7f7ffe0f6700 10 monclient: monmap: 2023-09-30T01:57:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:57:08.787 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:57:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:57:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:57:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:57:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:57:08.788 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.779+0000 7f7ffe0f6700 1 Processor -- start 2023-09-30T01:57:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.779+0000 7f7ffe0f6700 1 -- start start 2023-09-30T01:57:08.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.779+0000 7f7ffe0f6700 10 monclient: init 2023-09-30T01:57:08.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ffe0f6700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:08.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ffe0f6700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:57:08.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ffe0f6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ff8096a50 0x7f7ff812dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:08.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ffe0f6700 10 monclient(hunting): picked mon.noname-b con 0x7f7ff8096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:57:08.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ffe0f6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ff8097360 0x7f7ff809db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ffe0f6700 10 monclient(hunting): picked mon.noname-a con 0x7f7ff8097360 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:57:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ffe0f6700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7ff80a2390 0x7f7ff809e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ffe0f6700 10 monclient(hunting): picked mon.noname-c con 0x7f7ff80a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ffe0f6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ffe0f6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff81334e0 con 0x7f7ff8097360 2023-09-30T01:57:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ffe0f6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ffe0f6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff812f030 con 0x7f7ff8096a50 2023-09-30T01:57:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ffe0f6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ffe0f6700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff80a0290 con 0x7f7ff80a2390 2023-09-30T01:57:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ffe0f6700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:08.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ffe0f6700 10 monclient(hunting): authenticate will time out at 1317.845420s 2023-09-30T01:57:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ff8097360 0x7f7ff809db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff6ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ff8097360 0x7f7ff809db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37162/0 (socket says 172.21.15.7:37162) 2023-09-30T01:57:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff6ffd700 1 -- 172.21.15.7:0/3172331743 learned_addr learned my addr 172.21.15.7:0/3172331743 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff6ffd700 10 monclient(hunting): get_auth_request con 0x7f7ff8097360 auth_method 0 2023-09-30T01:57:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff7fff700 1 --2- 172.21.15.7:0/3172331743 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7ff80a2390 0x7f7ff809e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff7fff700 10 monclient(hunting): get_auth_request con 0x7f7ff80a2390 auth_method 0 2023-09-30T01:57:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:08.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:08.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:08.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:08.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:08.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:08.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:08.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:08.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff77fe700 1 --2- 172.21.15.7:0/3172331743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ff8096a50 0x7f7ff812dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:08.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff77fe700 10 monclient(hunting): get_auth_request con 0x7f7ff8096a50 auth_method 0 2023-09-30T01:57:08.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:08.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:08.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:08.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff7fff700 10 monclient(hunting): handle_auth_done global_id 4511 payload 824 2023-09-30T01:57:08.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff7fff700 1 -- 172.21.15.7:0/3172331743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ff8096a50 msgr2=0x7f7ff812dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:08.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff7fff700 1 --2- 172.21.15.7:0/3172331743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ff8096a50 0x7f7ff812dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:08.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff7fff700 1 -- 172.21.15.7:0/3172331743 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ff8097360 msgr2=0x7f7ff809db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:08.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff7fff700 1 --2- 172.21.15.7:0/3172331743 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ff8097360 0x7f7ff809db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:08.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff7fff700 10 monclient: _finish_hunting 0 2023-09-30T01:57:08.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff7fff700 1 monclient: found mon.noname-c 2023-09-30T01:57:08.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:08.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff7fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:57:08.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff7fff700 1 -- 172.21.15.7:0/3172331743 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fec009420 con 0x7f7ff80a2390 2023-09-30T01:57:08.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.780+0000 7f7ff6ffd700 1 --2- 172.21.15.7:0/3172331743 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ff8097360 0x7f7ff809db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:08.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7ff7fff700 1 -- 172.21.15.7:0/3172331743 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7ff80a2390 msgr2=0x7f7ff809e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:08.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7ff7fff700 1 --2- 172.21.15.7:0/3172331743 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7ff80a2390 0x7f7ff809e040 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7ff000cbc0 tx=0x7f7ff0005340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:08.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7fdf7fe700 1 -- 172.21.15.7:0/3172331743 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff000aa20 con 0x7f7ff80a2390 2023-09-30T01:57:08.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7fdf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:08.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7fdf7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:57:08.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7fdf7fe700 10 monclient: dump: 2023-09-30T01:57:08.798 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:08.798 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:08.798 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:08.798 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:08.798 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:08.798 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:08.799 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:08.799 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:08.799 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:08.799 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:08.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7fdf7fe700 10 monclient: _finish_auth 0 2023-09-30T01:57:08.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7fdf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:08.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7ffe0f6700 5 monclient: authenticate success, global_id 4511 2023-09-30T01:57:08.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7fdf7fe700 1 -- 172.21.15.7:0/3172331743 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff000aba0 con 0x7f7ff80a2390 2023-09-30T01:57:08.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7fdf7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:08.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7fdf7fe700 1 -- 172.21.15.7:0/3172331743 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff00070f0 con 0x7f7ff80a2390 2023-09-30T01:57:08.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7fdf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:08.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7fdf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:57:08.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7fdf7fe700 10 monclient: dump: 2023-09-30T01:57:08.800 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:08.800 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:08.800 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:08.800 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:08.800 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:08.800 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:08.800 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:08.800 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:08.800 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:08.801 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:08.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7ffe0f6700 10 monclient: _renew_subs 2023-09-30T01:57:08.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7ff5ffb700 4 set_mon_vals no callback set 2023-09-30T01:57:08.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7ffe0f6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:08.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7ffe0f6700 1 -- 172.21.15.7:0/3172331743 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff80a1180 con 0x7f7ff80a2390 2023-09-30T01:57:08.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7ffe0f6700 10 monclient: _renew_subs 2023-09-30T01:57:08.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7ffe0f6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:08.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.783+0000 7f7ffe0f6700 1 -- 172.21.15.7:0/3172331743 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff8142740 con 0x7f7ff80a2390 2023-09-30T01:57:08.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.784+0000 7f7fdf7fe700 1 -- 172.21.15.7:0/3172331743 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f7ff00074e0 con 0x7f7ff80a2390 2023-09-30T01:57:08.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.784+0000 7f7ffe0f6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:08.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.784+0000 7f7ffe0f6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:08.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.784+0000 7f7ffe0f6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:08.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.784+0000 7f7ffe0f6700 1 -- 172.21.15.7:0/3172331743 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff8064c10 con 0x7f7ff80a2390 2023-09-30T01:57:08.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.784+0000 7f7fdf7fe700 1 --2- 172.21.15.7:0/3172331743 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7fe40357e0 0x7f7fe4037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:08.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.784+0000 7f7fdf7fe700 1 -- 172.21.15.7:0/3172331743 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f7ff0048070 con 0x7f7ff80a2390 2023-09-30T01:57:08.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.784+0000 7f7ff77fe700 1 --2- 172.21.15.7:0/3172331743 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7fe40357e0 0x7f7fe4037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:08.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.784+0000 7f7ff77fe700 10 monclient: get_auth_request con 0x7f7fe40357e0 auth_method 0 2023-09-30T01:57:08.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.787+0000 7f7ff77fe700 1 --2- 172.21.15.7:0/3172331743 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7fe40357e0 0x7f7fe4037c90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8006f00 tx=0x7f7fe8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:08.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.789+0000 7f7fdf7fe700 1 -- 172.21.15.7:0/3172331743 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7ff0007790 con 0x7f7ff80a2390 2023-09-30T01:57:08.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.789+0000 7f7fdf7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:08.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:08.789+0000 7f7fdf7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:57:09.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:09.025+0000 7f7ffe0f6700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:57:09.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:09.025+0000 7f7ffe0f6700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:09.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:09.025+0000 7f7ffe0f6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:09.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:09.025+0000 7f7ffe0f6700 1 -- 172.21.15.7:0/3172331743 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7ff804bb40 con 0x7f7ff80a2390 2023-09-30T01:57:09.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:09.025+0000 7f7fdf7fe700 1 -- 172.21.15.7:0/3172331743 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7ff0010300 con 0x7f7ff80a2390 2023-09-30T01:57:09.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:09.025+0000 7f7fdf7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:09.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:09.025+0000 7f7fdf7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:57:09.026 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:57:09.026 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:57:09.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:09.027+0000 7f7ffe0f6700 1 -- 172.21.15.7:0/3172331743 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7fe40357e0 msgr2=0x7f7fe4037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:09.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:09.027+0000 7f7ffe0f6700 1 --2- 172.21.15.7:0/3172331743 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7fe40357e0 0x7f7fe4037c90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8006f00 tx=0x7f7fe8008040 comp rx=0 tx=0).stop 2023-09-30T01:57:09.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:09.027+0000 7f7ffe0f6700 10 monclient: shutdown 2023-09-30T01:57:09.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:09.027+0000 7f7ffe0f6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:09.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:09.027+0000 7f7ffe0f6700 1 -- 172.21.15.7:0/3172331743 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7ff80a2390 msgr2=0x7f7ff809e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:09.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:09.027+0000 7f7ffe0f6700 1 --2- 172.21.15.7:0/3172331743 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7ff80a2390 0x7f7ff809e040 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7ff000cbc0 tx=0x7f7ff0005340 comp rx=0 tx=0).stop 2023-09-30T01:57:09.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:09.027+0000 7f7ffe0f6700 1 -- 172.21.15.7:0/3172331743 shutdown_connections 2023-09-30T01:57:09.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:09.027+0000 7f7ffe0f6700 1 --2- 172.21.15.7:0/3172331743 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7fe40357e0 0x7f7fe4037c90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:09.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:09.027+0000 7f7ffe0f6700 1 --2- 172.21.15.7:0/3172331743 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7ff80a2390 0x7f7ff809e040 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:09.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:09.027+0000 7f7ffe0f6700 1 --2- 172.21.15.7:0/3172331743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ff8096a50 0x7f7ff812dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:09.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:09.027+0000 7f7ffe0f6700 1 --2- 172.21.15.7:0/3172331743 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ff8097360 0x7f7ff809db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:09.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:09.027+0000 7f7ffe0f6700 1 -- 172.21.15.7:0/3172331743 >> 172.21.15.7:0/3172331743 conn(0x7f7ff8070cf0 msgr2=0x7f7ff8130fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:09.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:09.028+0000 7f7ffe0f6700 1 -- 172.21.15.7:0/3172331743 shutdown_connections 2023-09-30T01:57:09.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:09.028+0000 7f7ffe0f6700 1 -- 172.21.15.7:0/3172331743 wait complete. 2023-09-30T01:57:12.040 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:57:12.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.166+0000 7ff3daa7d700 10 monclient: get_monmap_and_config 2023-09-30T01:57:12.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.166+0000 7ff3daa7d700 10 monclient: build_initial_monmap 2023-09-30T01:57:12.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.167+0000 7ff3daa7d700 10 monclient: monmap: 2023-09-30T01:57:12.167 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:12.167 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:12.167 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:57:12.168049+0000 2023-09-30T01:57:12.167 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:57:12.168049+0000 2023-09-30T01:57:12.168 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:12.168 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:12.168 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:57:12.168 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:57:12.168 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:57:12.168 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:57:12.168 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:57:12.168 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:12.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.167+0000 7ff3daa7d700 1 Processor -- start 2023-09-30T01:57:12.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.167+0000 7ff3daa7d700 1 -- start start 2023-09-30T01:57:12.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.167+0000 7ff3daa7d700 10 monclient: init 2023-09-30T01:57:12.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3daa7d700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:12.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3daa7d700 10 monclient: _add_conns ranks=[3,2,0,4,1] 2023-09-30T01:57:12.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3daa7d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff3d4096a50 0x7ff3d4132750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:12.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3daa7d700 10 monclient(hunting): picked mon.noname-a con 0x7ff3d4096a50 addr v2:172.21.15.7:3300/0 2023-09-30T01:57:12.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3daa7d700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff3d4096e20 0x7ff3d4097240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:12.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3daa7d700 10 monclient(hunting): picked mon.noname-b con 0x7ff3d4096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:12.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3daa7d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff3d412ef00 addr v1:172.21.15.7:6789/0 2023-09-30T01:57:12.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3daa7d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d4099480 con 0x7ff3d412ef00 2023-09-30T01:57:12.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3daa7d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:12.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3daa7d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3d4097bf0 con 0x7ff3d4096a50 2023-09-30T01:57:12.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3daa7d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:12.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3daa7d700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d40991a0 con 0x7ff3d4096e20 2023-09-30T01:57:12.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3daa7d700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:12.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3daa7d700 10 monclient(hunting): authenticate will time out at 1321.233261s 2023-09-30T01:57:12.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d8819700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff3d4096a50 0x7ff3d4132750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:12.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d901a700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff3d412ef00 0x7ff3d412f360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50034/0 (socket says 172.21.15.7:50034) 2023-09-30T01:57:12.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d8819700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff3d4096a50 0x7ff3d4132750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37170/0 (socket says 172.21.15.7:37170) 2023-09-30T01:57:12.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d901a700 1 -- 172.21.15.7:0/131002707 learned_addr learned my addr 172.21.15.7:0/131002707 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:57:12.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d3fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff3d4096e20 0x7ff3d4097240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:12.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d8819700 10 monclient(hunting): get_auth_request con 0x7ff3d4096a50 auth_method 0 2023-09-30T01:57:12.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d8819700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:12.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d8819700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:12.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d8819700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:12.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d3fff700 10 monclient(hunting): get_auth_request con 0x7ff3d4096e20 auth_method 0 2023-09-30T01:57:12.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:12.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:12.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:12.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:12.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:12.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:12.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d8819700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:12.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d8819700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:12.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d8819700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:12.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d3fff700 10 monclient(hunting): handle_auth_done global_id 4514 payload 274 2023-09-30T01:57:12.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d3fff700 1 -- 172.21.15.7:0/131002707 >> v2:172.21.15.7:3300/0 conn(0x7ff3d4096a50 msgr2=0x7ff3d4132750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:12.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d3fff700 1 --2- 172.21.15.7:0/131002707 >> v2:172.21.15.7:3300/0 conn(0x7ff3d4096a50 0x7ff3d4132750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:12.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d3fff700 1 -- 172.21.15.7:0/131002707 >> v1:172.21.15.7:6789/0 conn(0x7ff3d412ef00 legacy=0x7ff3d412f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:12.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d3fff700 10 monclient: _finish_hunting 0 2023-09-30T01:57:12.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d3fff700 1 monclient: found mon.noname-b 2023-09-30T01:57:12.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:12.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:57:12.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d3fff700 1 -- 172.21.15.7:0/131002707 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d4099760 con 0x7ff3d4096e20 2023-09-30T01:57:12.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.168+0000 7ff3d8819700 1 --2- 172.21.15.7:0/131002707 >> v2:172.21.15.7:3300/0 conn(0x7ff3d4096a50 0x7ff3d4132750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:12.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3d3fff700 1 -- 172.21.15.7:0/131002707 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff3d4096e20 msgr2=0x7ff3d4097240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:12.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3d3fff700 1 --2- 172.21.15.7:0/131002707 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff3d4096e20 0x7ff3d4097240 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff3c80098e0 tx=0x7ff3c80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=465764610da1f5ca server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:12.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3d37fe700 1 -- 172.21.15.7:0/131002707 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8011840 con 0x7ff3d4096e20 2023-09-30T01:57:12.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3d37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:12.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3d37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:57:12.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3d37fe700 10 monclient: dump: 2023-09-30T01:57:12.180 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:12.180 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:12.180 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:12.180 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:12.180 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:12.180 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:12.180 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:12.180 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:12.180 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:12.181 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:12.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3d37fe700 10 monclient: _finish_auth 0 2023-09-30T01:57:12.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3d37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:12.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3d37fe700 1 -- 172.21.15.7:0/131002707 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c80119a0 con 0x7ff3d4096e20 2023-09-30T01:57:12.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3daa7d700 5 monclient: authenticate success, global_id 4514 2023-09-30T01:57:12.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3daa7d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:57:12.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3d37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:12.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3daa7d700 10 monclient: get_monmap_and_config success 2023-09-30T01:57:12.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3d37fe700 1 -- 172.21.15.7:0/131002707 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8011d40 con 0x7ff3d4096e20 2023-09-30T01:57:12.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3daa7d700 4 set_mon_vals no callback set 2023-09-30T01:57:12.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3d37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:12.181 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3d37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:57:12.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3d37fe700 10 monclient: dump: 2023-09-30T01:57:12.182 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:12.182 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:12.182 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:12.182 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:12.182 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:12.182 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:12.182 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:12.182 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:12.182 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:12.182 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:12.182 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3daa7d700 10 monclient: shutdown 2023-09-30T01:57:12.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3daa7d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:12.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3daa7d700 1 -- 172.21.15.7:0/131002707 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff3d4096e20 msgr2=0x7ff3d4097240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:12.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3daa7d700 1 --2- 172.21.15.7:0/131002707 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff3d4096e20 0x7ff3d4097240 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff3c80098e0 tx=0x7ff3c80091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:12.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3daa7d700 1 -- 172.21.15.7:0/131002707 shutdown_connections 2023-09-30T01:57:12.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3daa7d700 1 --2- 172.21.15.7:0/131002707 >> v2:172.21.15.7:3300/0 conn(0x7ff3d4096a50 0x7ff3d4132750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:12.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3daa7d700 1 --2- 172.21.15.7:0/131002707 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff3d4096e20 0x7ff3d4097240 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:12.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3daa7d700 1 -- 172.21.15.7:0/131002707 >> 172.21.15.7:0/131002707 conn(0x7ff3d4070cf0 msgr2=0x7ff3d40710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:12.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3daa7d700 1 -- 172.21.15.7:0/131002707 shutdown_connections 2023-09-30T01:57:12.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.171+0000 7ff3daa7d700 1 -- 172.21.15.7:0/131002707 wait complete. 2023-09-30T01:57:12.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.172+0000 7ff3daa7d700 10 monclient: build_initial_monmap 2023-09-30T01:57:12.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.172+0000 7ff3daa7d700 10 monclient: monmap: 2023-09-30T01:57:12.184 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:12.184 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:57:12.184 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:57:12.184 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:57:12.184 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:12.184 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:12.184 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:57:12.184 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:57:12.184 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:57:12.184 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:12.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.172+0000 7ff3daa7d700 1 Processor -- start 2023-09-30T01:57:12.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.172+0000 7ff3daa7d700 1 -- start start 2023-09-30T01:57:12.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.172+0000 7ff3daa7d700 10 monclient: init 2023-09-30T01:57:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3daa7d700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3daa7d700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:57:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3daa7d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff3d4096a50 0x7ff3d412def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3daa7d700 10 monclient(hunting): picked mon.noname-b con 0x7ff3d4096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:57:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3daa7d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3d4096e20 0x7ff3d409d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3daa7d700 10 monclient(hunting): picked mon.noname-a con 0x7ff3d4096e20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:57:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3daa7d700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff3d412ef00 0x7ff3d409df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3daa7d700 10 monclient(hunting): picked mon.noname-c con 0x7ff3d412ef00 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3daa7d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3daa7d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d40a1770 con 0x7ff3d4096e20 2023-09-30T01:57:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3daa7d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:12.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3daa7d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d40a5420 con 0x7ff3d4096a50 2023-09-30T01:57:12.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3daa7d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:12.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3daa7d700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d40a11b0 con 0x7ff3d412ef00 2023-09-30T01:57:12.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3daa7d700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:12.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3daa7d700 10 monclient(hunting): authenticate will time out at 1321.238487s 2023-09-30T01:57:12.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3d4096e20 0x7ff3d409d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:12.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3d4096e20 0x7ff3d409d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37180/0 (socket says 172.21.15.7:37180) 2023-09-30T01:57:12.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d3fff700 1 -- 172.21.15.7:0/1783933223 learned_addr learned my addr 172.21.15.7:0/1783933223 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:12.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d901a700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff3d412ef00 0x7ff3d409df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:12.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d3fff700 10 monclient(hunting): get_auth_request con 0x7ff3d4096e20 auth_method 0 2023-09-30T01:57:12.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d901a700 10 monclient(hunting): get_auth_request con 0x7ff3d412ef00 auth_method 0 2023-09-30T01:57:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d901a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d901a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d901a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d8819700 1 --2- 172.21.15.7:0/1783933223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff3d4096a50 0x7ff3d412def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d901a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d901a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:12.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d901a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:12.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d8819700 10 monclient(hunting): get_auth_request con 0x7ff3d4096a50 auth_method 0 2023-09-30T01:57:12.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d8819700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:12.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d8819700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:12.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d8819700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:12.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d8819700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:12.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d8819700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:12.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d8819700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:12.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d3fff700 10 monclient(hunting): handle_auth_done global_id 4503 payload 824 2023-09-30T01:57:12.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d3fff700 1 -- 172.21.15.7:0/1783933223 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff3d412ef00 msgr2=0x7ff3d409df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:12.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d3fff700 1 --2- 172.21.15.7:0/1783933223 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff3d412ef00 0x7ff3d409df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:12.188 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d3fff700 1 -- 172.21.15.7:0/1783933223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff3d4096a50 msgr2=0x7ff3d412def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:12.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d3fff700 1 --2- 172.21.15.7:0/1783933223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff3d4096a50 0x7ff3d412def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:12.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d3fff700 10 monclient: _finish_hunting 0 2023-09-30T01:57:12.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d3fff700 1 monclient: found mon.noname-a 2023-09-30T01:57:12.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:12.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d3fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:57:12.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d3fff700 1 -- 172.21.15.7:0/1783933223 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c4009420 con 0x7ff3d4096e20 2023-09-30T01:57:12.189 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.173+0000 7ff3d901a700 1 --2- 172.21.15.7:0/1783933223 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff3d412ef00 0x7ff3d409df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:12.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3d3fff700 1 -- 172.21.15.7:0/1783933223 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3d4096e20 msgr2=0x7ff3d409d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:12.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3d3fff700 1 --2- 172.21.15.7:0/1783933223 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3d4096e20 0x7ff3d409d9d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff3c80093f0 tx=0x7ff3c8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:12.191 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3d0ff9700 1 -- 172.21.15.7:0/1783933223 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c801b470 con 0x7ff3d4096e20 2023-09-30T01:57:12.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3d0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:12.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3d0ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:57:12.192 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3d0ff9700 10 monclient: dump: 2023-09-30T01:57:12.192 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:12.192 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:12.192 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:12.192 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:12.192 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:12.192 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:12.192 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:12.192 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:12.193 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:12.193 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:12.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3d0ff9700 10 monclient: _finish_auth 0 2023-09-30T01:57:12.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3d0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:12.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3daa7d700 5 monclient: authenticate success, global_id 4503 2023-09-30T01:57:12.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3d0ff9700 1 -- 172.21.15.7:0/1783933223 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c801b5d0 con 0x7ff3d4096e20 2023-09-30T01:57:12.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3d0ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:12.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3daa7d700 10 monclient: _renew_subs 2023-09-30T01:57:12.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3d0ff9700 1 -- 172.21.15.7:0/1783933223 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c801b990 con 0x7ff3d4096e20 2023-09-30T01:57:12.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3daa7d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:12.193 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3daa7d700 1 -- 172.21.15.7:0/1783933223 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c8009420 con 0x7ff3d4096e20 2023-09-30T01:57:12.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3d0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:12.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3d2ffd700 4 set_mon_vals no callback set 2023-09-30T01:57:12.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3d0ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:57:12.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3d0ff9700 10 monclient: dump: 2023-09-30T01:57:12.194 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:12.194 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:12.194 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:12.194 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:12.194 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:12.194 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:12.194 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:12.195 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:12.195 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:12.195 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:12.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3daa7d700 10 monclient: _renew_subs 2023-09-30T01:57:12.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3daa7d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:12.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.176+0000 7ff3daa7d700 1 -- 172.21.15.7:0/1783933223 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d409f680 con 0x7ff3d4096e20 2023-09-30T01:57:12.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.177+0000 7ff3d0ff9700 1 -- 172.21.15.7:0/1783933223 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7ff3c801c450 con 0x7ff3d4096e20 2023-09-30T01:57:12.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.177+0000 7ff3daa7d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:12.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.177+0000 7ff3daa7d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:12.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.177+0000 7ff3daa7d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:12.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.177+0000 7ff3daa7d700 1 -- 172.21.15.7:0/1783933223 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d40a46c0 con 0x7ff3d4096e20 2023-09-30T01:57:12.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.177+0000 7ff3d0ff9700 1 --2- 172.21.15.7:0/1783933223 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff3c00357e0 0x7ff3c0037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:12.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.177+0000 7ff3d0ff9700 1 -- 172.21.15.7:0/1783933223 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7ff3c8049770 con 0x7ff3d4096e20 2023-09-30T01:57:12.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.177+0000 7ff3d8819700 1 --2- 172.21.15.7:0/1783933223 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff3c00357e0 0x7ff3c0037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:12.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.177+0000 7ff3d8819700 10 monclient: get_auth_request con 0x7ff3c00357e0 auth_method 0 2023-09-30T01:57:12.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.180+0000 7ff3d8819700 1 --2- 172.21.15.7:0/1783933223 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff3c00357e0 0x7ff3c0037c90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4009f40 tx=0x7ff3c40091a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:12.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.182+0000 7ff3d0ff9700 1 -- 172.21.15.7:0/1783933223 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff3c8018350 con 0x7ff3d4096e20 2023-09-30T01:57:12.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.182+0000 7ff3d0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:12.196 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.182+0000 7ff3d0ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:57:12.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.418+0000 7ff3daa7d700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:57:12.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.418+0000 7ff3daa7d700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:12.418 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.418+0000 7ff3daa7d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:12.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.418+0000 7ff3daa7d700 1 -- 172.21.15.7:0/1783933223 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff3d40a15a0 con 0x7ff3d4096e20 2023-09-30T01:57:12.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.418+0000 7ff3d0ff9700 1 -- 172.21.15.7:0/1783933223 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff3c802a020 con 0x7ff3d4096e20 2023-09-30T01:57:12.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.418+0000 7ff3d0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:12.419 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.418+0000 7ff3d0ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:57:12.419 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:57:12.419 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:57:12.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.420+0000 7ff3daa7d700 1 -- 172.21.15.7:0/1783933223 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff3c00357e0 msgr2=0x7ff3c0037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:12.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.420+0000 7ff3daa7d700 1 --2- 172.21.15.7:0/1783933223 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff3c00357e0 0x7ff3c0037c90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4009f40 tx=0x7ff3c40091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:12.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.420+0000 7ff3daa7d700 10 monclient: shutdown 2023-09-30T01:57:12.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.420+0000 7ff3daa7d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:12.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.420+0000 7ff3daa7d700 1 -- 172.21.15.7:0/1783933223 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3d4096e20 msgr2=0x7ff3d409d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:12.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.420+0000 7ff3daa7d700 1 --2- 172.21.15.7:0/1783933223 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3d4096e20 0x7ff3d409d9d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff3c80093f0 tx=0x7ff3c8016000 comp rx=0 tx=0).stop 2023-09-30T01:57:12.421 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.421+0000 7ff3daa7d700 1 -- 172.21.15.7:0/1783933223 shutdown_connections 2023-09-30T01:57:12.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.421+0000 7ff3daa7d700 1 --2- 172.21.15.7:0/1783933223 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff3c00357e0 0x7ff3c0037c90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:12.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.421+0000 7ff3daa7d700 1 --2- 172.21.15.7:0/1783933223 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff3d412ef00 0x7ff3d409df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:12.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.421+0000 7ff3daa7d700 1 --2- 172.21.15.7:0/1783933223 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff3d4096a50 0x7ff3d412def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:12.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.421+0000 7ff3daa7d700 1 --2- 172.21.15.7:0/1783933223 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3d4096e20 0x7ff3d409d9d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:12.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.421+0000 7ff3daa7d700 1 -- 172.21.15.7:0/1783933223 >> 172.21.15.7:0/1783933223 conn(0x7ff3d4070cf0 msgr2=0x7ff3d4135860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:12.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.421+0000 7ff3daa7d700 1 -- 172.21.15.7:0/1783933223 shutdown_connections 2023-09-30T01:57:12.422 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:12.421+0000 7ff3daa7d700 1 -- 172.21.15.7:0/1783933223 wait complete. 2023-09-30T01:57:15.433 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:57:15.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.563+0000 7f79c0697700 10 monclient: get_monmap_and_config 2023-09-30T01:57:15.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.563+0000 7f79c0697700 10 monclient: build_initial_monmap 2023-09-30T01:57:15.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.564+0000 7f79c0697700 10 monclient: monmap: 2023-09-30T01:57:15.565 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:15.565 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:15.565 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:57:15.565025+0000 2023-09-30T01:57:15.565 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:57:15.565025+0000 2023-09-30T01:57:15.565 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:15.565 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:15.565 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:57:15.565 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:57:15.565 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:57:15.565 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:57:15.565 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:57:15.565 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:15.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.564+0000 7f79c0697700 1 Processor -- start 2023-09-30T01:57:15.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.564+0000 7f79c0697700 1 -- start start 2023-09-30T01:57:15.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.564+0000 7f79c0697700 10 monclient: init 2023-09-30T01:57:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79c0697700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79c0697700 10 monclient: _add_conns ranks=[0,1,2,4,3] 2023-09-30T01:57:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79c0697700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f79b80a29e0 addr v1:172.21.15.7:6789/0 2023-09-30T01:57:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79c0697700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f79b8097160 addr v1:172.21.15.50:6789/0 2023-09-30T01:57:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79c0697700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79b8097560 0x7f79b813ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79c0697700 10 monclient(hunting): picked mon.noname-b con 0x7f79b8097560 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79c0697700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b8069cc0 con 0x7f79b80a29e0 2023-09-30T01:57:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79c0697700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b80a2db0 con 0x7f79b8097160 2023-09-30T01:57:15.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79c0697700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:15.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79c0697700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b8098ea0 con 0x7f79b8097560 2023-09-30T01:57:15.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79c0697700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:15.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79c0697700 10 monclient(hunting): authenticate will time out at 1324.630471s 2023-09-30T01:57:15.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79be433700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f79b80a29e0 0x7f79b8096a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56208/0 (socket says 172.21.15.7:56208) 2023-09-30T01:57:15.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79bec34700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79b8097560 0x7f79b813ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:15.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79be433700 1 -- 172.21.15.7:0/2311158582 learned_addr learned my addr 172.21.15.7:0/2311158582 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:57:15.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79bec34700 10 monclient(hunting): get_auth_request con 0x7f79b8097560 auth_method 0 2023-09-30T01:57:15.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79bec34700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:15.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79bec34700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:15.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79bec34700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:15.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79bec34700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:15.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79bec34700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:15.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79bec34700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:15.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79bec34700 10 monclient(hunting): handle_auth_done global_id 4520 payload 274 2023-09-30T01:57:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79bec34700 1 -- 172.21.15.7:0/2311158582 >> v1:172.21.15.50:6789/0 conn(0x7f79b8097160 legacy=0x7f79b8137550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79bec34700 1 -- 172.21.15.7:0/2311158582 >> v1:172.21.15.7:6789/0 conn(0x7f79b80a29e0 legacy=0x7f79b8096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79bec34700 10 monclient: _finish_hunting 0 2023-09-30T01:57:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79bec34700 1 monclient: found mon.noname-b 2023-09-30T01:57:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79bec34700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79bec34700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:57:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.565+0000 7f79bec34700 1 -- 172.21.15.7:0/2311158582 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b8098270 con 0x7f79b8097560 2023-09-30T01:57:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79bec34700 1 -- 172.21.15.7:0/2311158582 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79b8097560 msgr2=0x7f79b813ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79bec34700 1 --2- 172.21.15.7:0/2311158582 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79b8097560 0x7f79b813ac80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f79b0009e90 tx=0x7f79b0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19e77618ee574a73 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79bd431700 1 -- 172.21.15.7:0/2311158582 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b000e070 con 0x7f79b8097560 2023-09-30T01:57:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79bd431700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79bd431700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:57:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79bd431700 10 monclient: dump: 2023-09-30T01:57:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:15.573 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:15.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79bd431700 10 monclient: _finish_auth 0 2023-09-30T01:57:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79bd431700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79bd431700 1 -- 172.21.15.7:0/2311158582 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b0012430 con 0x7f79b8097560 2023-09-30T01:57:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79c0697700 5 monclient: authenticate success, global_id 4520 2023-09-30T01:57:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79c0697700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:57:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79bd431700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79c0697700 10 monclient: get_monmap_and_config success 2023-09-30T01:57:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79c0697700 4 set_mon_vals no callback set 2023-09-30T01:57:15.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79bd431700 1 -- 172.21.15.7:0/2311158582 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b000bea0 con 0x7f79b8097560 2023-09-30T01:57:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79bd431700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79bd431700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:57:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79bd431700 10 monclient: dump: 2023-09-30T01:57:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:15.576 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:15.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79c0697700 10 monclient: shutdown 2023-09-30T01:57:15.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79c0697700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:15.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79c0697700 1 -- 172.21.15.7:0/2311158582 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79b8097560 msgr2=0x7f79b813ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:15.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79c0697700 1 --2- 172.21.15.7:0/2311158582 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79b8097560 0x7f79b813ac80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f79b0009e90 tx=0x7f79b0009780 comp rx=0 tx=0).stop 2023-09-30T01:57:15.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79c0697700 1 -- 172.21.15.7:0/2311158582 shutdown_connections 2023-09-30T01:57:15.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79c0697700 1 --2- 172.21.15.7:0/2311158582 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79b8097560 0x7f79b813ac80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:15.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79c0697700 1 -- 172.21.15.7:0/2311158582 >> 172.21.15.7:0/2311158582 conn(0x7f79b80710d0 msgr2=0x7f79b80714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:15.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79c0697700 1 -- 172.21.15.7:0/2311158582 shutdown_connections 2023-09-30T01:57:15.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.568+0000 7f79c0697700 1 -- 172.21.15.7:0/2311158582 wait complete. 2023-09-30T01:57:15.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.569+0000 7f79c0697700 10 monclient: build_initial_monmap 2023-09-30T01:57:15.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.569+0000 7f79c0697700 10 monclient: monmap: 2023-09-30T01:57:15.577 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:15.577 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:57:15.577 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:57:15.577 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:57:15.577 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:15.577 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:15.577 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:57:15.577 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:57:15.577 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:57:15.578 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:15.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.569+0000 7f79c0697700 1 Processor -- start 2023-09-30T01:57:15.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.569+0000 7f79c0697700 1 -- start start 2023-09-30T01:57:15.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.569+0000 7f79c0697700 10 monclient: init 2023-09-30T01:57:15.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79c0697700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:15.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79c0697700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:57:15.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79c0697700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79b8097160 0x7f79b812e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:15.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79c0697700 10 monclient(hunting): picked mon.noname-b con 0x7f79b8097160 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:57:15.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79c0697700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79b8097560 0x7f79b81416c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:15.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79c0697700 10 monclient(hunting): picked mon.noname-c con 0x7f79b8097560 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:15.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79c0697700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79b80a29e0 0x7f79b8141c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:15.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79c0697700 10 monclient(hunting): picked mon.noname-a con 0x7f79b80a29e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:57:15.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79c0697700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:15.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79c0697700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b80a0250 con 0x7f79b80a29e0 2023-09-30T01:57:15.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79c0697700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:15.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79c0697700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b811ca80 con 0x7f79b8097160 2023-09-30T01:57:15.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79c0697700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:15.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79c0697700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b80a1220 con 0x7f79b8097560 2023-09-30T01:57:15.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79c0697700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:15.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79c0697700 10 monclient(hunting): authenticate will time out at 1324.635350s 2023-09-30T01:57:15.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bdc32700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79b8097560 0x7f79b81416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:15.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bec34700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79b80a29e0 0x7f79b8141c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:15.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bec34700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79b80a29e0 0x7f79b8141c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59894/0 (socket says 172.21.15.7:59894) 2023-09-30T01:57:15.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bdc32700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79b8097560 0x7f79b81416c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:51134/0 (socket says 172.21.15.7:51134) 2023-09-30T01:57:15.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bec34700 1 -- 172.21.15.7:0/1421304094 learned_addr learned my addr 172.21.15.7:0/1421304094 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:15.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bec34700 10 monclient(hunting): get_auth_request con 0x7f79b80a29e0 auth_method 0 2023-09-30T01:57:15.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bec34700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:15.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bec34700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:15.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bec34700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:15.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bdc32700 10 monclient(hunting): get_auth_request con 0x7f79b8097560 auth_method 0 2023-09-30T01:57:15.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bdc32700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:15.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bdc32700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:15.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bdc32700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:15.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bec34700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:15.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bec34700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:15.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bec34700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:15.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bdc32700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:15.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bdc32700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:15.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bdc32700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:15.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79be433700 1 --2- 172.21.15.7:0/1421304094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79b8097160 0x7f79b812e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:15.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79be433700 10 monclient(hunting): get_auth_request con 0x7f79b8097160 auth_method 0 2023-09-30T01:57:15.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79be433700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:15.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79be433700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:15.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79be433700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:15.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bdc32700 10 monclient(hunting): handle_auth_done global_id 4523 payload 824 2023-09-30T01:57:15.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bdc32700 1 -- 172.21.15.7:0/1421304094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79b8097160 msgr2=0x7f79b812e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:15.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bdc32700 1 --2- 172.21.15.7:0/1421304094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79b8097160 0x7f79b812e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:15.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bdc32700 1 -- 172.21.15.7:0/1421304094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79b80a29e0 msgr2=0x7f79b8141c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:15.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bdc32700 1 --2- 172.21.15.7:0/1421304094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79b80a29e0 0x7f79b8141c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:15.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bdc32700 10 monclient: _finish_hunting 0 2023-09-30T01:57:15.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bdc32700 1 monclient: found mon.noname-c 2023-09-30T01:57:15.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bdc32700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:15.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bdc32700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:57:15.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bdc32700 1 -- 172.21.15.7:0/1421304094 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b0009a20 con 0x7f79b8097560 2023-09-30T01:57:15.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79bec34700 1 --2- 172.21.15.7:0/1421304094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79b80a29e0 0x7f79b8141c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:15.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.570+0000 7f79be433700 1 --2- 172.21.15.7:0/1421304094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79b8097160 0x7f79b812e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:57:15.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79bdc32700 1 -- 172.21.15.7:0/1421304094 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79b8097560 msgr2=0x7f79b81416c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:15.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79bdc32700 1 --2- 172.21.15.7:0/1421304094 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79b8097560 0x7f79b81416c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f79a800c540 tx=0x7f79a8007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:15.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79ae7fc700 1 -- 172.21.15.7:0/1421304094 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a80117a0 con 0x7f79b8097560 2023-09-30T01:57:15.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79ae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:15.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79ae7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:57:15.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79ae7fc700 10 monclient: dump: 2023-09-30T01:57:15.587 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:15.588 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:15.588 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:15.588 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:15.588 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:15.588 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:15.588 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:15.588 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:15.588 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:15.588 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:15.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79ae7fc700 10 monclient: _finish_auth 0 2023-09-30T01:57:15.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79ae7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:15.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79c0697700 5 monclient: authenticate success, global_id 4523 2023-09-30T01:57:15.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79ae7fc700 1 -- 172.21.15.7:0/1421304094 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a8011920 con 0x7f79b8097560 2023-09-30T01:57:15.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79ae7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:15.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79ae7fc700 1 -- 172.21.15.7:0/1421304094 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a8011ce0 con 0x7f79b8097560 2023-09-30T01:57:15.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79c0697700 10 monclient: _renew_subs 2023-09-30T01:57:15.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79c0697700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:15.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79c0697700 1 -- 172.21.15.7:0/1421304094 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b811c7a0 con 0x7f79b8097560 2023-09-30T01:57:15.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79bcc30700 4 set_mon_vals no callback set 2023-09-30T01:57:15.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79ae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:15.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79ae7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:57:15.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79ae7fc700 10 monclient: dump: 2023-09-30T01:57:15.589 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:15.589 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:15.590 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:15.590 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:15.590 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:15.590 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:15.590 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:15.590 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:15.590 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:15.590 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:15.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79c0697700 10 monclient: _renew_subs 2023-09-30T01:57:15.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79c0697700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:15.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.573+0000 7f79c0697700 1 -- 172.21.15.7:0/1421304094 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b809fcb0 con 0x7f79b8097560 2023-09-30T01:57:15.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.574+0000 7f79ae7fc700 1 -- 172.21.15.7:0/1421304094 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f79a8012630 con 0x7f79b8097560 2023-09-30T01:57:15.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.574+0000 7f79ae7fc700 1 --2- 172.21.15.7:0/1421304094 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f799c0357e0 0x7f799c037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:15.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.574+0000 7f79ae7fc700 1 -- 172.21.15.7:0/1421304094 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f79a80479a0 con 0x7f79b8097560 2023-09-30T01:57:15.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.574+0000 7f79c0697700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:15.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.574+0000 7f79c0697700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:15.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.574+0000 7f79c0697700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:15.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.574+0000 7f79c0697700 1 -- 172.21.15.7:0/1421304094 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b8064c10 con 0x7f79b8097560 2023-09-30T01:57:15.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.574+0000 7f79be433700 1 --2- 172.21.15.7:0/1421304094 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f799c0357e0 0x7f799c037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:15.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.574+0000 7f79be433700 10 monclient: get_auth_request con 0x7f799c0357e0 auth_method 0 2023-09-30T01:57:15.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.577+0000 7f79be433700 1 --2- 172.21.15.7:0/1421304094 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f799c0357e0 0x7f799c037c90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f79b4007fd0 tx=0x7f79b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:15.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.579+0000 7f79ae7fc700 1 -- 172.21.15.7:0/1421304094 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f79a8002090 con 0x7f79b8097560 2023-09-30T01:57:15.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.579+0000 7f79ae7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:15.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.579+0000 7f79ae7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:57:15.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.817+0000 7f79c0697700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:57:15.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.817+0000 7f79c0697700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:15.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.817+0000 7f79c0697700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:15.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.817+0000 7f79c0697700 1 -- 172.21.15.7:0/1421304094 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f79b811cbc0 con 0x7f79b8097560 2023-09-30T01:57:15.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.818+0000 7f79ae7fc700 1 -- 172.21.15.7:0/1421304094 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f79a80102e0 con 0x7f79b8097560 2023-09-30T01:57:15.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.818+0000 7f79ae7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:15.818 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.818+0000 7f79ae7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:57:15.819 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:57:15.819 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:57:15.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.820+0000 7f79c0697700 1 -- 172.21.15.7:0/1421304094 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f799c0357e0 msgr2=0x7f799c037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:15.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.820+0000 7f79c0697700 1 --2- 172.21.15.7:0/1421304094 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f799c0357e0 0x7f799c037c90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f79b4007fd0 tx=0x7f79b4009040 comp rx=0 tx=0).stop 2023-09-30T01:57:15.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.820+0000 7f79c0697700 10 monclient: shutdown 2023-09-30T01:57:15.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.820+0000 7f79c0697700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:15.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.820+0000 7f79c0697700 1 -- 172.21.15.7:0/1421304094 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79b8097560 msgr2=0x7f79b81416c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:15.820 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.820+0000 7f79c0697700 1 --2- 172.21.15.7:0/1421304094 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79b8097560 0x7f79b81416c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f79a800c540 tx=0x7f79a8007ea0 comp rx=0 tx=0).stop 2023-09-30T01:57:15.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.820+0000 7f79c0697700 1 -- 172.21.15.7:0/1421304094 shutdown_connections 2023-09-30T01:57:15.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.820+0000 7f79c0697700 1 --2- 172.21.15.7:0/1421304094 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f799c0357e0 0x7f799c037c90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:15.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.820+0000 7f79c0697700 1 --2- 172.21.15.7:0/1421304094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79b80a29e0 0x7f79b8141c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:15.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.820+0000 7f79c0697700 1 --2- 172.21.15.7:0/1421304094 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79b8097160 0x7f79b812e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:15.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.820+0000 7f79c0697700 1 --2- 172.21.15.7:0/1421304094 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79b8097560 0x7f79b81416c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:15.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.820+0000 7f79c0697700 1 -- 172.21.15.7:0/1421304094 >> 172.21.15.7:0/1421304094 conn(0x7f79b80710d0 msgr2=0x7f79b8130510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:15.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.820+0000 7f79c0697700 1 -- 172.21.15.7:0/1421304094 shutdown_connections 2023-09-30T01:57:15.821 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:15.820+0000 7f79c0697700 1 -- 172.21.15.7:0/1421304094 wait complete. 2023-09-30T01:57:18.832 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:57:18.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.963+0000 7fbb74769700 10 monclient: get_monmap_and_config 2023-09-30T01:57:18.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.963+0000 7fbb74769700 10 monclient: build_initial_monmap 2023-09-30T01:57:18.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.963+0000 7fbb74769700 10 monclient: monmap: 2023-09-30T01:57:18.964 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:18.964 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:18.964 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:57:18.964193+0000 2023-09-30T01:57:18.964 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:57:18.964193+0000 2023-09-30T01:57:18.964 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:18.964 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:18.964 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:57:18.964 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:57:18.964 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:57:18.964 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:57:18.964 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:57:18.965 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:18.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.963+0000 7fbb74769700 1 Processor -- start 2023-09-30T01:57:18.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.963+0000 7fbb74769700 1 -- start start 2023-09-30T01:57:18.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.963+0000 7fbb74769700 10 monclient: init 2023-09-30T01:57:18.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb74769700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:18.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb74769700 10 monclient: _add_conns ranks=[4,0,3,1,2] 2023-09-30T01:57:18.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb74769700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fbb6c096a50 0x7fbb6c132680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:18.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb74769700 10 monclient(hunting): picked mon.noname-c con 0x7fbb6c096a50 addr v2:172.21.15.50:3300/0 2023-09-30T01:57:18.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb74769700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbb6c096e20 addr v1:172.21.15.7:6789/0 2023-09-30T01:57:18.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb74769700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbb6c137710 0x7fbb6c139af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:18.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb74769700 10 monclient(hunting): picked mon.noname-a con 0x7fbb6c137710 addr v2:172.21.15.7:3300/0 2023-09-30T01:57:18.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb74769700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6c099440 con 0x7fbb6c096e20 2023-09-30T01:57:18.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb74769700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:18.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb74769700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb6c097b40 con 0x7fbb6c137710 2023-09-30T01:57:18.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb74769700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:18.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb74769700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb6c099160 con 0x7fbb6c096a50 2023-09-30T01:57:18.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb74769700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:18.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb74769700 10 monclient(hunting): authenticate will time out at 1328.029473s 2023-09-30T01:57:18.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb72d06700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbb6c137710 0x7fbb6c139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:18.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb72d06700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbb6c137710 0x7fbb6c139af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59914/0 (socket says 172.21.15.7:59914) 2023-09-30T01:57:18.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb72d06700 1 -- 172.21.15.7:0/308886964 learned_addr learned my addr 172.21.15.7:0/308886964 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:18.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb72d06700 10 monclient(hunting): get_auth_request con 0x7fbb6c137710 auth_method 0 2023-09-30T01:57:18.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb72d06700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:18.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb72d06700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:18.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb72d06700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:18.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb71d04700 1 --1- 172.21.15.7:0/308886964 >> v1:172.21.15.7:6789/0 conn(0x7fbb6c096e20 0x7fbb6c097220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56212/0 (socket says 172.21.15.7:56212) 2023-09-30T01:57:18.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb72505700 1 --2- 172.21.15.7:0/308886964 >> v2:172.21.15.50:3300/0 conn(0x7fbb6c096a50 0x7fbb6c132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:18.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb72505700 10 monclient(hunting): get_auth_request con 0x7fbb6c096a50 auth_method 0 2023-09-30T01:57:18.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb72505700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:18.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb72505700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:18.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb72505700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:18.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb72505700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:18.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb72505700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:18.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.964+0000 7fbb72505700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:18.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.965+0000 7fbb72505700 10 monclient(hunting): handle_auth_done global_id 4462 payload 274 2023-09-30T01:57:18.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.965+0000 7fbb72505700 1 -- 172.21.15.7:0/308886964 >> v2:172.21.15.7:3300/0 conn(0x7fbb6c137710 msgr2=0x7fbb6c139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:18.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.965+0000 7fbb72505700 1 --2- 172.21.15.7:0/308886964 >> v2:172.21.15.7:3300/0 conn(0x7fbb6c137710 0x7fbb6c139af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:18.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.965+0000 7fbb72505700 1 -- 172.21.15.7:0/308886964 >> v1:172.21.15.7:6789/0 conn(0x7fbb6c096e20 legacy=0x7fbb6c097220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:18.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.965+0000 7fbb72505700 10 monclient: _finish_hunting 0 2023-09-30T01:57:18.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.965+0000 7fbb72505700 1 monclient: found mon.noname-c 2023-09-30T01:57:18.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.965+0000 7fbb72505700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:18.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.965+0000 7fbb72505700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3300/0 2023-09-30T01:57:18.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.965+0000 7fbb72505700 1 -- 172.21.15.7:0/308886964 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6c099720 con 0x7fbb6c096a50 2023-09-30T01:57:18.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.967+0000 7fbb72505700 1 -- 172.21.15.7:0/308886964 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6c096a50 msgr2=0x7fbb6c132680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:18.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.967+0000 7fbb72505700 1 --2- 172.21.15.7:0/308886964 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6c096a50 0x7fbb6c132680 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0a2b30 tx=0x7fbb68009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16f03acc8f87fdcb server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:18.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.967+0000 7fbb71503700 1 -- 172.21.15.7:0/308886964 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6800e070 con 0x7fbb6c096a50 2023-09-30T01:57:18.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.967+0000 7fbb71503700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:18.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.967+0000 7fbb71503700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:57:18.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.967+0000 7fbb71503700 10 monclient: dump: 2023-09-30T01:57:18.976 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:18.976 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:18.976 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:18.976 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:18.976 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:18.976 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:18.976 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:18.976 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:18.976 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:18.976 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb71503700 10 monclient: _finish_auth 0 2023-09-30T01:57:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb71503700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb71503700 1 -- 172.21.15.7:0/308886964 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6800be00 con 0x7fbb6c096a50 2023-09-30T01:57:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb74769700 5 monclient: authenticate success, global_id 4462 2023-09-30T01:57:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb74769700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:57:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb71503700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb74769700 10 monclient: get_monmap_and_config success 2023-09-30T01:57:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb71503700 1 -- 172.21.15.7:0/308886964 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb68012660 con 0x7fbb6c096a50 2023-09-30T01:57:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb71503700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb74769700 4 set_mon_vals no callback set 2023-09-30T01:57:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb71503700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:57:18.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb71503700 10 monclient: dump: 2023-09-30T01:57:18.978 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:18.978 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:18.978 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:18.978 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:18.978 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:18.978 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:18.978 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:18.978 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:18.978 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:18.978 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:18.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb74769700 10 monclient: shutdown 2023-09-30T01:57:18.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb74769700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:18.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb74769700 1 -- 172.21.15.7:0/308886964 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6c096a50 msgr2=0x7fbb6c132680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:18.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb74769700 1 --2- 172.21.15.7:0/308886964 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6c096a50 0x7fbb6c132680 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0a2b30 tx=0x7fbb68009780 comp rx=0 tx=0).stop 2023-09-30T01:57:18.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb74769700 1 -- 172.21.15.7:0/308886964 shutdown_connections 2023-09-30T01:57:18.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb74769700 1 --2- 172.21.15.7:0/308886964 >> v2:172.21.15.7:3300/0 conn(0x7fbb6c137710 0x7fbb6c139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:18.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb74769700 1 --2- 172.21.15.7:0/308886964 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6c096a50 0x7fbb6c132680 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:18.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb74769700 1 -- 172.21.15.7:0/308886964 >> 172.21.15.7:0/308886964 conn(0x7fbb6c070cf0 msgr2=0x7fbb6c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:18.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb74769700 1 -- 172.21.15.7:0/308886964 shutdown_connections 2023-09-30T01:57:18.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.968+0000 7fbb74769700 1 -- 172.21.15.7:0/308886964 wait complete. 2023-09-30T01:57:18.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.969+0000 7fbb74769700 10 monclient: build_initial_monmap 2023-09-30T01:57:18.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.969+0000 7fbb74769700 10 monclient: monmap: 2023-09-30T01:57:18.983 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:18.983 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:57:18.983 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:57:18.983 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:57:18.983 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:18.983 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:18.983 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:57:18.983 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:57:18.984 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:57:18.984 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:18.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.969+0000 7fbb74769700 1 Processor -- start 2023-09-30T01:57:18.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.969+0000 7fbb74769700 1 -- start start 2023-09-30T01:57:18.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.969+0000 7fbb74769700 10 monclient: init 2023-09-30T01:57:18.984 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.969+0000 7fbb74769700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.969+0000 7fbb74769700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:57:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.969+0000 7fbb74769700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb6c096a50 0x7fbb6c09cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.969+0000 7fbb74769700 10 monclient(hunting): picked mon.noname-a con 0x7fbb6c096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:57:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.969+0000 7fbb74769700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbb6c096e20 0x7fbb6c09d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.969+0000 7fbb74769700 10 monclient(hunting): picked mon.noname-c con 0x7fbb6c096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.969+0000 7fbb74769700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6c137710 0x7fbb6c09d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.969+0000 7fbb74769700 10 monclient(hunting): picked mon.noname-b con 0x7fbb6c137710 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:57:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.969+0000 7fbb74769700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.969+0000 7fbb74769700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c09ec40 con 0x7fbb6c096a50 2023-09-30T01:57:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.969+0000 7fbb74769700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.969+0000 7fbb74769700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c09ff40 con 0x7fbb6c137710 2023-09-30T01:57:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.969+0000 7fbb74769700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.969+0000 7fbb74769700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c1334e0 con 0x7fbb6c096e20 2023-09-30T01:57:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.969+0000 7fbb74769700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.969+0000 7fbb74769700 10 monclient(hunting): authenticate will time out at 1328.034941s 2023-09-30T01:57:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72505700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb6c096a50 0x7fbb6c09cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72505700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb6c096a50 0x7fbb6c09cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59918/0 (socket says 172.21.15.7:59918) 2023-09-30T01:57:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72505700 1 -- 172.21.15.7:0/2956206924 learned_addr learned my addr 172.21.15.7:0/2956206924 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72505700 10 monclient(hunting): get_auth_request con 0x7fbb6c096a50 auth_method 0 2023-09-30T01:57:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72505700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72505700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72505700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb71d04700 1 --2- 172.21.15.7:0/2956206924 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbb6c096e20 0x7fbb6c09d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb71d04700 10 monclient(hunting): get_auth_request con 0x7fbb6c096e20 auth_method 0 2023-09-30T01:57:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb71d04700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb71d04700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb71d04700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:18.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72d06700 1 --2- 172.21.15.7:0/2956206924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6c137710 0x7fbb6c09d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:18.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72505700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:18.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72505700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:18.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72505700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:18.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72d06700 10 monclient(hunting): get_auth_request con 0x7fbb6c137710 auth_method 0 2023-09-30T01:57:18.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72d06700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:18.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72d06700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:18.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72d06700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:18.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb71d04700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:18.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb71d04700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:18.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb71d04700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:18.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72d06700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:18.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72d06700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:18.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72d06700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:18.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72505700 10 monclient(hunting): handle_auth_done global_id 4512 payload 824 2023-09-30T01:57:18.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72505700 1 -- 172.21.15.7:0/2956206924 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbb6c096e20 msgr2=0x7fbb6c09d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72505700 1 --2- 172.21.15.7:0/2956206924 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbb6c096e20 0x7fbb6c09d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72505700 1 -- 172.21.15.7:0/2956206924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6c137710 msgr2=0x7fbb6c09d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72505700 1 --2- 172.21.15.7:0/2956206924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6c137710 0x7fbb6c09d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72505700 10 monclient: _finish_hunting 0 2023-09-30T01:57:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72505700 1 monclient: found mon.noname-a 2023-09-30T01:57:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72505700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72505700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:57:18.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb72505700 1 -- 172.21.15.7:0/2956206924 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb68009a20 con 0x7fbb6c096a50 2023-09-30T01:57:18.992 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.970+0000 7fbb71d04700 1 --2- 172.21.15.7:0/2956206924 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbb6c096e20 0x7fbb6c09d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:18.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb72505700 1 -- 172.21.15.7:0/2956206924 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb6c096a50 msgr2=0x7fbb6c09cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:18.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb72505700 1 --2- 172.21.15.7:0/2956206924 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb6c096a50 0x7fbb6c09cbe0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbb68009fd0 tx=0x7fbb68018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:18.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb627fc700 1 -- 172.21.15.7:0/2956206924 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb68012cb0 con 0x7fbb6c096a50 2023-09-30T01:57:18.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:18.995 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb627fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:57:18.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb627fc700 10 monclient: dump: 2023-09-30T01:57:18.996 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:18.996 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:18.996 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:18.996 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:18.996 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:18.997 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:18.997 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:18.997 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:18.997 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:18.997 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:18.997 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb627fc700 10 monclient: _finish_auth 0 2023-09-30T01:57:18.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb627fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:18.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb627fc700 1 -- 172.21.15.7:0/2956206924 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb68021070 con 0x7fbb6c096a50 2023-09-30T01:57:18.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb74769700 5 monclient: authenticate success, global_id 4512 2023-09-30T01:57:18.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb74769700 10 monclient: _renew_subs 2023-09-30T01:57:18.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb74769700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:18.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb74769700 1 -- 172.21.15.7:0/2956206924 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb6c131cf0 con 0x7fbb6c096a50 2023-09-30T01:57:18.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb627fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:18.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb627fc700 1 -- 172.21.15.7:0/2956206924 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb68026680 con 0x7fbb6c096a50 2023-09-30T01:57:18.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:18.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb627fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:57:18.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb627fc700 10 monclient: dump: 2023-09-30T01:57:19.000 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:19.000 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:19.000 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:19.000 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:19.000 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:19.001 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:19.001 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:19.001 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:19.001 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:19.001 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:19.001 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb70d02700 4 set_mon_vals no callback set 2023-09-30T01:57:19.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb74769700 10 monclient: _renew_subs 2023-09-30T01:57:19.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb74769700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:19.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.972+0000 7fbb74769700 1 -- 172.21.15.7:0/2956206924 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb6c0a0230 con 0x7fbb6c096a50 2023-09-30T01:57:19.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.973+0000 7fbb627fc700 1 -- 172.21.15.7:0/2956206924 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fbb68023030 con 0x7fbb6c096a50 2023-09-30T01:57:19.002 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.973+0000 7fbb74769700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:19.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.973+0000 7fbb74769700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:19.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.973+0000 7fbb74769700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:19.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.973+0000 7fbb74769700 1 -- 172.21.15.7:0/2956206924 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb6c064c10 con 0x7fbb6c096a50 2023-09-30T01:57:19.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.973+0000 7fbb627fc700 1 --2- 172.21.15.7:0/2956206924 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fbb500357e0 0x7fbb50037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:19.003 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.974+0000 7fbb627fc700 1 -- 172.21.15.7:0/2956206924 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fbb68049c80 con 0x7fbb6c096a50 2023-09-30T01:57:19.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.974+0000 7fbb71d04700 1 --2- 172.21.15.7:0/2956206924 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fbb500357e0 0x7fbb50037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:19.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.974+0000 7fbb71d04700 10 monclient: get_auth_request con 0x7fbb500357e0 auth_method 0 2023-09-30T01:57:19.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.976+0000 7fbb71d04700 1 --2- 172.21.15.7:0/2956206924 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fbb500357e0 0x7fbb50037c90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0a6000 tx=0x7fbb5c007ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:19.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.978+0000 7fbb627fc700 1 -- 172.21.15.7:0/2956206924 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbb68012e30 con 0x7fbb6c096a50 2023-09-30T01:57:19.004 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.978+0000 7fbb627fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:19.005 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:18.978+0000 7fbb627fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:57:19.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:19.214+0000 7fbb74769700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:57:19.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:19.214+0000 7fbb74769700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:19.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:19.214+0000 7fbb74769700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:19.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:19.214+0000 7fbb74769700 1 -- 172.21.15.7:0/2956206924 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbb6c0a2440 con 0x7fbb6c096a50 2023-09-30T01:57:19.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:19.215+0000 7fbb627fc700 1 -- 172.21.15.7:0/2956206924 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fbb680253d0 con 0x7fbb6c096a50 2023-09-30T01:57:19.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:19.215+0000 7fbb627fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:19.215 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:19.215+0000 7fbb627fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:57:19.215 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:57:19.215 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:57:19.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:19.217+0000 7fbb74769700 1 -- 172.21.15.7:0/2956206924 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fbb500357e0 msgr2=0x7fbb50037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:19.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:19.217+0000 7fbb74769700 1 --2- 172.21.15.7:0/2956206924 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fbb500357e0 0x7fbb50037c90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0a6000 tx=0x7fbb5c007ed0 comp rx=0 tx=0).stop 2023-09-30T01:57:19.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:19.217+0000 7fbb74769700 10 monclient: shutdown 2023-09-30T01:57:19.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:19.217+0000 7fbb74769700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:19.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:19.217+0000 7fbb74769700 1 -- 172.21.15.7:0/2956206924 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb6c096a50 msgr2=0x7fbb6c09cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:19.217 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:19.217+0000 7fbb74769700 1 --2- 172.21.15.7:0/2956206924 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb6c096a50 0x7fbb6c09cbe0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbb68009fd0 tx=0x7fbb68018000 comp rx=0 tx=0).stop 2023-09-30T01:57:19.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:19.217+0000 7fbb74769700 1 -- 172.21.15.7:0/2956206924 shutdown_connections 2023-09-30T01:57:19.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:19.217+0000 7fbb74769700 1 --2- 172.21.15.7:0/2956206924 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fbb500357e0 0x7fbb50037c90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:19.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:19.217+0000 7fbb74769700 1 --2- 172.21.15.7:0/2956206924 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbb6c096a50 0x7fbb6c09cbe0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:19.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:19.217+0000 7fbb74769700 1 --2- 172.21.15.7:0/2956206924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6c137710 0x7fbb6c09d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:19.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:19.217+0000 7fbb74769700 1 --2- 172.21.15.7:0/2956206924 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbb6c096e20 0x7fbb6c09d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:19.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:19.217+0000 7fbb74769700 1 -- 172.21.15.7:0/2956206924 >> 172.21.15.7:0/2956206924 conn(0x7fbb6c070cf0 msgr2=0x7fbb6c12f110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:19.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:19.217+0000 7fbb74769700 1 -- 172.21.15.7:0/2956206924 shutdown_connections 2023-09-30T01:57:19.218 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:19.217+0000 7fbb74769700 1 -- 172.21.15.7:0/2956206924 wait complete. 2023-09-30T01:57:22.231 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:57:22.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.362+0000 7f7a3a4aa700 10 monclient: get_monmap_and_config 2023-09-30T01:57:22.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.362+0000 7f7a3a4aa700 10 monclient: build_initial_monmap 2023-09-30T01:57:22.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.362+0000 7f7a3a4aa700 10 monclient: monmap: 2023-09-30T01:57:22.363 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:22.363 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:22.363 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:57:22.363606+0000 2023-09-30T01:57:22.363 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:57:22.363606+0000 2023-09-30T01:57:22.363 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:22.363 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:22.363 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:57:22.363 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:57:22.363 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:57:22.364 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:57:22.364 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:57:22.364 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:22.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.363+0000 7f7a3a4aa700 1 Processor -- start 2023-09-30T01:57:22.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.363+0000 7f7a3a4aa700 1 -- start start 2023-09-30T01:57:22.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.363+0000 7f7a3a4aa700 10 monclient: init 2023-09-30T01:57:22.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.363+0000 7f7a3a4aa700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:22.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.363+0000 7f7a3a4aa700 10 monclient: _add_conns ranks=[0,4,2,1,3] 2023-09-30T01:57:22.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.363+0000 7f7a3a4aa700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a34096a50 addr v1:172.21.15.7:6789/0 2023-09-30T01:57:22.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.363+0000 7f7a3a4aa700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f7a34096e20 0x7f7a34097220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:22.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.363+0000 7f7a3a4aa700 10 monclient(hunting): picked mon.noname-c con 0x7f7a34096e20 addr v2:172.21.15.50:3300/0 2023-09-30T01:57:22.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.363+0000 7f7a3a4aa700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a34137710 0x7f7a34139af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:22.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.363+0000 7f7a3a4aa700 10 monclient(hunting): picked mon.noname-b con 0x7f7a34137710 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:22.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.363+0000 7f7a3a4aa700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a34099470 con 0x7f7a34096a50 2023-09-30T01:57:22.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.363+0000 7f7a3a4aa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:22.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.363+0000 7f7a3a4aa700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a34132d90 con 0x7f7a34096e20 2023-09-30T01:57:22.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.363+0000 7f7a3a4aa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:22.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.363+0000 7f7a3a4aa700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a34097b40 con 0x7f7a34137710 2023-09-30T01:57:22.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.363+0000 7f7a3a4aa700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:22.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.363+0000 7f7a3a4aa700 10 monclient(hunting): authenticate will time out at 1331.428836s 2023-09-30T01:57:22.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.363+0000 7f7a38a47700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a34137710 0x7f7a34139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:22.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.363+0000 7f7a33fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7a34096a50 0x7f7a34132680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56222/0 (socket says 172.21.15.7:56222) 2023-09-30T01:57:22.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.363+0000 7f7a33fff700 1 -- 172.21.15.7:0/3247942177 learned_addr learned my addr 172.21.15.7:0/3247942177 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:57:22.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.364+0000 7f7a38a47700 10 monclient(hunting): get_auth_request con 0x7f7a34137710 auth_method 0 2023-09-30T01:57:22.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.364+0000 7f7a38a47700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:22.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.364+0000 7f7a38a47700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:22.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.364+0000 7f7a38a47700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:22.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.364+0000 7f7a38a47700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:22.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.364+0000 7f7a38a47700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:22.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.364+0000 7f7a38a47700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:22.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.364+0000 7f7a38a47700 10 monclient(hunting): handle_auth_done global_id 4529 payload 274 2023-09-30T01:57:22.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.364+0000 7f7a38a47700 1 -- 172.21.15.7:0/3247942177 >> v2:172.21.15.50:3300/0 conn(0x7f7a34096e20 msgr2=0x7f7a34097220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:22.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.364+0000 7f7a38a47700 1 --2- 172.21.15.7:0/3247942177 >> v2:172.21.15.50:3300/0 conn(0x7f7a34096e20 0x7f7a34097220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:22.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.364+0000 7f7a38a47700 1 -- 172.21.15.7:0/3247942177 >> v1:172.21.15.7:6789/0 conn(0x7f7a34096a50 legacy=0x7f7a34132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:22.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.364+0000 7f7a38a47700 10 monclient: _finish_hunting 0 2023-09-30T01:57:22.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.364+0000 7f7a38a47700 1 monclient: found mon.noname-b 2023-09-30T01:57:22.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.364+0000 7f7a38a47700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:22.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.364+0000 7f7a38a47700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:57:22.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.364+0000 7f7a38a47700 1 -- 172.21.15.7:0/3247942177 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a34099160 con 0x7f7a34137710 2023-09-30T01:57:22.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.366+0000 7f7a38a47700 1 -- 172.21.15.7:0/3247942177 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a34137710 msgr2=0x7f7a34139af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:22.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.366+0000 7f7a38a47700 1 --2- 172.21.15.7:0/3247942177 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a34137710 0x7f7a34139af0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c009e60 tx=0x7f7a2c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10869efd2a45bebc server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:22.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.366+0000 7f7a32ffd700 1 -- 172.21.15.7:0/3247942177 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c00e070 con 0x7f7a34137710 2023-09-30T01:57:22.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.366+0000 7f7a32ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:22.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.366+0000 7f7a32ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:57:22.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.366+0000 7f7a32ffd700 10 monclient: dump: 2023-09-30T01:57:22.376 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:22.376 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:22.376 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:22.376 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:22.376 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:22.376 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:22.377 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:22.377 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:22.377 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:22.377 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:22.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.366+0000 7f7a32ffd700 10 monclient: _finish_auth 0 2023-09-30T01:57:22.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.366+0000 7f7a32ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:22.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.366+0000 7f7a32ffd700 1 -- 172.21.15.7:0/3247942177 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2c012430 con 0x7f7a34137710 2023-09-30T01:57:22.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.366+0000 7f7a3a4aa700 5 monclient: authenticate success, global_id 4529 2023-09-30T01:57:22.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.366+0000 7f7a3a4aa700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:57:22.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.366+0000 7f7a32ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:22.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.366+0000 7f7a32ffd700 1 -- 172.21.15.7:0/3247942177 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c0127d0 con 0x7f7a34137710 2023-09-30T01:57:22.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.366+0000 7f7a3a4aa700 10 monclient: get_monmap_and_config success 2023-09-30T01:57:22.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.366+0000 7f7a3a4aa700 4 set_mon_vals no callback set 2023-09-30T01:57:22.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.366+0000 7f7a32ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:22.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.366+0000 7f7a32ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:57:22.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.366+0000 7f7a32ffd700 10 monclient: dump: 2023-09-30T01:57:22.378 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:22.378 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:22.378 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:22.378 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:22.378 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:22.378 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:22.378 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:22.378 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:22.379 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:22.379 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:22.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.367+0000 7f7a3a4aa700 10 monclient: shutdown 2023-09-30T01:57:22.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.367+0000 7f7a3a4aa700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:22.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.367+0000 7f7a3a4aa700 1 -- 172.21.15.7:0/3247942177 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a34137710 msgr2=0x7f7a34139af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:22.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.367+0000 7f7a3a4aa700 1 --2- 172.21.15.7:0/3247942177 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a34137710 0x7f7a34139af0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c009e60 tx=0x7f7a2c009780 comp rx=0 tx=0).stop 2023-09-30T01:57:22.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.367+0000 7f7a3a4aa700 1 -- 172.21.15.7:0/3247942177 shutdown_connections 2023-09-30T01:57:22.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.367+0000 7f7a3a4aa700 1 --2- 172.21.15.7:0/3247942177 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a34137710 0x7f7a34139af0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:22.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.367+0000 7f7a3a4aa700 1 --2- 172.21.15.7:0/3247942177 >> v2:172.21.15.50:3300/0 conn(0x7f7a34096e20 0x7f7a34097220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:22.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.367+0000 7f7a3a4aa700 1 -- 172.21.15.7:0/3247942177 >> 172.21.15.7:0/3247942177 conn(0x7f7a34070cf0 msgr2=0x7f7a340710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:22.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.367+0000 7f7a3a4aa700 1 -- 172.21.15.7:0/3247942177 shutdown_connections 2023-09-30T01:57:22.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.367+0000 7f7a3a4aa700 1 -- 172.21.15.7:0/3247942177 wait complete. 2023-09-30T01:57:22.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.367+0000 7f7a3a4aa700 10 monclient: build_initial_monmap 2023-09-30T01:57:22.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.367+0000 7f7a3a4aa700 10 monclient: monmap: 2023-09-30T01:57:22.380 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:22.380 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:57:22.380 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:57:22.380 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:57:22.380 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:22.380 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:22.380 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:57:22.380 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:57:22.380 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:57:22.380 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:22.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.368+0000 7f7a3a4aa700 1 Processor -- start 2023-09-30T01:57:22.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.368+0000 7f7a3a4aa700 1 -- start start 2023-09-30T01:57:22.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.368+0000 7f7a3a4aa700 10 monclient: init 2023-09-30T01:57:22.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.368+0000 7f7a3a4aa700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:22.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.368+0000 7f7a3a4aa700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:57:22.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.368+0000 7f7a3a4aa700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a34096a50 0x7f7a34099f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:22.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.368+0000 7f7a3a4aa700 10 monclient(hunting): picked mon.noname-a con 0x7f7a34096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:57:22.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.368+0000 7f7a3a4aa700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a34096e20 0x7f7a3409a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:22.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.368+0000 7f7a3a4aa700 10 monclient(hunting): picked mon.noname-c con 0x7f7a34096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:22.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.368+0000 7f7a3a4aa700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a34137710 0x7f7a3409a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:22.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.368+0000 7f7a3a4aa700 10 monclient(hunting): picked mon.noname-b con 0x7f7a34137710 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:57:22.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.368+0000 7f7a3a4aa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:22.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.368+0000 7f7a3a4aa700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a34131a30 con 0x7f7a34096a50 2023-09-30T01:57:22.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.368+0000 7f7a3a4aa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:22.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.368+0000 7f7a3a4aa700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a341334e0 con 0x7f7a34137710 2023-09-30T01:57:22.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.368+0000 7f7a3a4aa700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:22.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.368+0000 7f7a3a4aa700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3409cec0 con 0x7f7a34096e20 2023-09-30T01:57:22.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.368+0000 7f7a3a4aa700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:22.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.368+0000 7f7a3a4aa700 10 monclient(hunting): authenticate will time out at 1331.433924s 2023-09-30T01:57:22.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.368+0000 7f7a337fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a34096e20 0x7f7a3409a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:22.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a33fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a34096a50 0x7f7a34099f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:22.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a33fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a34096a50 0x7f7a34099f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:59920/0 (socket says 172.21.15.7:59920) 2023-09-30T01:57:22.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a337fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a34096e20 0x7f7a3409a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:51180/0 (socket says 172.21.15.7:51180) 2023-09-30T01:57:22.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a33fff700 1 -- 172.21.15.7:0/1265492159 learned_addr learned my addr 172.21.15.7:0/1265492159 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:22.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a33fff700 10 monclient(hunting): get_auth_request con 0x7f7a34096a50 auth_method 0 2023-09-30T01:57:22.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a33fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:22.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a33fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:22.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a33fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:22.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a337fe700 10 monclient(hunting): get_auth_request con 0x7f7a34096e20 auth_method 0 2023-09-30T01:57:22.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:22.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a337fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:22.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a337fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:22.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a38a47700 1 --2- 172.21.15.7:0/1265492159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a34137710 0x7f7a3409a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:22.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:22.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:22.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:22.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a33fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:22.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a33fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:22.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a33fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:22.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a38a47700 10 monclient(hunting): get_auth_request con 0x7f7a34137710 auth_method 0 2023-09-30T01:57:22.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a38a47700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:22.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a38a47700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:22.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a38a47700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:22.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a38a47700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:22.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a38a47700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:22.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a38a47700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:22.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a337fe700 10 monclient(hunting): handle_auth_done global_id 4532 payload 824 2023-09-30T01:57:22.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a337fe700 1 -- 172.21.15.7:0/1265492159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a34137710 msgr2=0x7f7a3409a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:22.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a337fe700 1 --2- 172.21.15.7:0/1265492159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a34137710 0x7f7a3409a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:22.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a337fe700 1 -- 172.21.15.7:0/1265492159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a34096a50 msgr2=0x7f7a34099f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:22.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a337fe700 1 --2- 172.21.15.7:0/1265492159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a34096a50 0x7f7a34099f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:22.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a337fe700 10 monclient: _finish_hunting 0 2023-09-30T01:57:22.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a337fe700 1 monclient: found mon.noname-c 2023-09-30T01:57:22.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a337fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:22.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a337fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:57:22.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a337fe700 1 -- 172.21.15.7:0/1265492159 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c009a20 con 0x7f7a34096e20 2023-09-30T01:57:22.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.369+0000 7f7a33fff700 1 --2- 172.21.15.7:0/1265492159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a34096a50 0x7f7a34099f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:22.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a337fe700 1 -- 172.21.15.7:0/1265492159 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a34096e20 msgr2=0x7f7a3409a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:22.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a337fe700 1 --2- 172.21.15.7:0/1265492159 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a34096e20 0x7f7a3409a460 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7a2800b630 tx=0x7f7a280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:22.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a1bfff700 1 -- 172.21.15.7:0/1265492159 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2800f720 con 0x7f7a34096e20 2023-09-30T01:57:22.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a1bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:22.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a1bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:57:22.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a1bfff700 10 monclient: dump: 2023-09-30T01:57:22.386 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:22.386 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:22.386 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:22.386 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:22.386 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:22.386 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:22.386 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:22.386 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:22.387 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:22.387 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:22.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a1bfff700 10 monclient: _finish_auth 0 2023-09-30T01:57:22.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a1bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:22.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a3a4aa700 5 monclient: authenticate success, global_id 4532 2023-09-30T01:57:22.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a1bfff700 1 -- 172.21.15.7:0/1265492159 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2800f8a0 con 0x7f7a34096e20 2023-09-30T01:57:22.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a1bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:22.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a1bfff700 1 -- 172.21.15.7:0/1265492159 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2800fc60 con 0x7f7a34096e20 2023-09-30T01:57:22.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a3a4aa700 10 monclient: _renew_subs 2023-09-30T01:57:22.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a3a4aa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:22.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a3a4aa700 1 -- 172.21.15.7:0/1265492159 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a3409b390 con 0x7f7a34096e20 2023-09-30T01:57:22.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a327fc700 4 set_mon_vals no callback set 2023-09-30T01:57:22.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a1bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:22.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a1bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:57:22.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a1bfff700 10 monclient: dump: 2023-09-30T01:57:22.388 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:22.388 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:22.388 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:22.388 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:22.388 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:22.388 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:22.388 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:22.388 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:22.388 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:22.389 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:22.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a3a4aa700 10 monclient: _renew_subs 2023-09-30T01:57:22.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a3a4aa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:22.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.371+0000 7f7a3a4aa700 1 -- 172.21.15.7:0/1265492159 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a3409bfd0 con 0x7f7a34096e20 2023-09-30T01:57:22.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.372+0000 7f7a1bfff700 1 -- 172.21.15.7:0/1265492159 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f7a28011610 con 0x7f7a34096e20 2023-09-30T01:57:22.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.373+0000 7f7a3a4aa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:22.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.373+0000 7f7a3a4aa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:22.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.373+0000 7f7a3a4aa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:22.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.373+0000 7f7a3a4aa700 1 -- 172.21.15.7:0/1265492159 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a3409d360 con 0x7f7a34096e20 2023-09-30T01:57:22.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.373+0000 7f7a1bfff700 1 --2- 172.21.15.7:0/1265492159 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7a200357e0 0x7f7a20037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:22.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.373+0000 7f7a1bfff700 1 -- 172.21.15.7:0/1265492159 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f7a28047820 con 0x7f7a34096e20 2023-09-30T01:57:22.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.373+0000 7f7a33fff700 1 --2- 172.21.15.7:0/1265492159 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7a200357e0 0x7f7a20037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:22.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.373+0000 7f7a33fff700 10 monclient: get_auth_request con 0x7f7a200357e0 auth_method 0 2023-09-30T01:57:22.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.375+0000 7f7a33fff700 1 --2- 172.21.15.7:0/1265492159 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7a200357e0 0x7f7a20037c90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7a240052e0 tx=0x7f7a24004a80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:22.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.378+0000 7f7a1bfff700 1 -- 172.21.15.7:0/1265492159 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7a280178a0 con 0x7f7a34096e20 2023-09-30T01:57:22.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.378+0000 7f7a1bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:22.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.378+0000 7f7a1bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:57:22.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.616+0000 7f7a3a4aa700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:57:22.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.616+0000 7f7a3a4aa700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:22.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.616+0000 7f7a3a4aa700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:22.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.616+0000 7f7a3a4aa700 1 -- 172.21.15.7:0/1265492159 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7a34064c10 con 0x7f7a34096e20 2023-09-30T01:57:22.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.617+0000 7f7a1bfff700 1 -- 172.21.15.7:0/1265492159 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7a280189c0 con 0x7f7a34096e20 2023-09-30T01:57:22.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.617+0000 7f7a1bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:22.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.617+0000 7f7a1bfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:57:22.618 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:57:22.618 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:57:22.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.619+0000 7f7a3a4aa700 1 -- 172.21.15.7:0/1265492159 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7a200357e0 msgr2=0x7f7a20037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:22.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.619+0000 7f7a3a4aa700 1 --2- 172.21.15.7:0/1265492159 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7a200357e0 0x7f7a20037c90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7a240052e0 tx=0x7f7a24004a80 comp rx=0 tx=0).stop 2023-09-30T01:57:22.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.619+0000 7f7a3a4aa700 10 monclient: shutdown 2023-09-30T01:57:22.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.619+0000 7f7a3a4aa700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:22.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.619+0000 7f7a3a4aa700 1 -- 172.21.15.7:0/1265492159 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a34096e20 msgr2=0x7f7a3409a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:22.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.619+0000 7f7a3a4aa700 1 --2- 172.21.15.7:0/1265492159 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a34096e20 0x7f7a3409a460 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7a2800b630 tx=0x7f7a280091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:22.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.619+0000 7f7a3a4aa700 1 -- 172.21.15.7:0/1265492159 shutdown_connections 2023-09-30T01:57:22.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.619+0000 7f7a3a4aa700 1 --2- 172.21.15.7:0/1265492159 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7a200357e0 0x7f7a20037c90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:22.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.619+0000 7f7a3a4aa700 1 --2- 172.21.15.7:0/1265492159 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7a34096a50 0x7f7a34099f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:22.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.619+0000 7f7a3a4aa700 1 --2- 172.21.15.7:0/1265492159 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a34137710 0x7f7a3409a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:22.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.619+0000 7f7a3a4aa700 1 --2- 172.21.15.7:0/1265492159 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f7a34096e20 0x7f7a3409a460 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:22.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.619+0000 7f7a3a4aa700 1 -- 172.21.15.7:0/1265492159 >> 172.21.15.7:0/1265492159 conn(0x7f7a34070cf0 msgr2=0x7f7a340a4500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:22.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.619+0000 7f7a3a4aa700 1 -- 172.21.15.7:0/1265492159 shutdown_connections 2023-09-30T01:57:22.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:22.619+0000 7f7a3a4aa700 1 -- 172.21.15.7:0/1265492159 wait complete. 2023-09-30T01:57:25.631 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:57:25.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.763+0000 7f390933b700 10 monclient: get_monmap_and_config 2023-09-30T01:57:25.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.763+0000 7f390933b700 10 monclient: build_initial_monmap 2023-09-30T01:57:25.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.763+0000 7f390933b700 10 monclient: monmap: 2023-09-30T01:57:25.763 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:25.763 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:25.763 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:57:25.764361+0000 2023-09-30T01:57:25.764 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:57:25.764361+0000 2023-09-30T01:57:25.764 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:25.764 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:25.764 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:57:25.764 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:57:25.764 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:57:25.764 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:57:25.764 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:57:25.764 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.763+0000 7f390933b700 1 Processor -- start 2023-09-30T01:57:25.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.763+0000 7f390933b700 1 -- start start 2023-09-30T01:57:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.763+0000 7f390933b700 10 monclient: init 2023-09-30T01:57:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f390933b700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f390933b700 10 monclient: _add_conns ranks=[1,2,4,0,3] 2023-09-30T01:57:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f390933b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3904096a50 addr v1:172.21.15.50:6789/0 2023-09-30T01:57:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f390933b700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39040a2390 0x7f3904096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f390933b700 10 monclient(hunting): picked mon.noname-b con 0x7f39040a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f390933b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3904097360 0x7f3904139720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f390933b700 10 monclient(hunting): picked mon.noname-c con 0x7f3904097360 addr v2:172.21.15.50:3300/0 2023-09-30T01:57:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f390933b700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39040980e0 con 0x7f3904096a50 2023-09-30T01:57:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f390933b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:25.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f390933b700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3904132eb0 con 0x7f3904097360 2023-09-30T01:57:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f390933b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f390933b700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39040991a0 con 0x7f39040a2390 2023-09-30T01:57:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f390933b700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f390933b700 10 monclient(hunting): authenticate will time out at 1334.829566s 2023-09-30T01:57:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f39027fc700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39040a2390 0x7f3904096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f39027fc700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39040a2390 0x7f3904096e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38510/0 (socket says 172.21.15.7:38510) 2023-09-30T01:57:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f39027fc700 1 -- 172.21.15.7:0/2897866567 learned_addr learned my addr 172.21.15.7:0/2897866567 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f39027fc700 10 monclient(hunting): get_auth_request con 0x7f39040a2390 auth_method 0 2023-09-30T01:57:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f39027fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f39027fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:25.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f39027fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:25.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f39037fe700 1 --2- 172.21.15.7:0/2897866567 >> v2:172.21.15.50:3300/0 conn(0x7f3904097360 0x7f3904139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:25.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f39027fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:25.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f39027fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:25.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f39027fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:25.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f39037fe700 10 monclient(hunting): get_auth_request con 0x7f3904097360 auth_method 0 2023-09-30T01:57:25.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f39037fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:25.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f39037fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:25.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f39037fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:25.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f39037fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:25.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f39037fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:25.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f39037fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:25.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f39027fc700 10 monclient(hunting): handle_auth_done global_id 4535 payload 274 2023-09-30T01:57:25.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f39027fc700 1 -- 172.21.15.7:0/2897866567 >> v2:172.21.15.50:3300/0 conn(0x7f3904097360 msgr2=0x7f3904139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:25.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.764+0000 7f39027fc700 1 --2- 172.21.15.7:0/2897866567 >> v2:172.21.15.50:3300/0 conn(0x7f3904097360 0x7f3904139720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:25.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.765+0000 7f39027fc700 1 -- 172.21.15.7:0/2897866567 >> v1:172.21.15.50:6789/0 conn(0x7f3904096a50 legacy=0x7f39041327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:25.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.765+0000 7f39027fc700 10 monclient: _finish_hunting 0 2023-09-30T01:57:25.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.765+0000 7f39027fc700 1 monclient: found mon.noname-b 2023-09-30T01:57:25.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.765+0000 7f39027fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:25.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.765+0000 7f39027fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:57:25.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.765+0000 7f39027fc700 1 -- 172.21.15.7:0/2897866567 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3904099480 con 0x7f39040a2390 2023-09-30T01:57:25.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f39027fc700 1 -- 172.21.15.7:0/2897866567 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39040a2390 msgr2=0x7f3904096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:25.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f39027fc700 1 --2- 172.21.15.7:0/2897866567 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39040a2390 0x7f3904096e20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f38ec0099b0 tx=0x7f38ec009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99fab69a4dec41a6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:25.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f3901ffb700 1 -- 172.21.15.7:0/2897866567 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec009ea0 con 0x7f39040a2390 2023-09-30T01:57:25.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f3901ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:25.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f3901ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:57:25.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f3901ffb700 10 monclient: dump: 2023-09-30T01:57:25.775 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:25.775 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:25.775 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:25.775 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:25.775 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:25.776 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:25.776 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:25.776 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:25.776 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:25.776 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:25.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f3901ffb700 10 monclient: _finish_auth 0 2023-09-30T01:57:25.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f3901ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:25.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f3901ffb700 1 -- 172.21.15.7:0/2897866567 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ec00ba30 con 0x7f39040a2390 2023-09-30T01:57:25.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f390933b700 5 monclient: authenticate success, global_id 4535 2023-09-30T01:57:25.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f390933b700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:57:25.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f3901ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:25.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f390933b700 10 monclient: get_monmap_and_config success 2023-09-30T01:57:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f3901ffb700 1 -- 172.21.15.7:0/2897866567 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec00bdf0 con 0x7f39040a2390 2023-09-30T01:57:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f390933b700 4 set_mon_vals no callback set 2023-09-30T01:57:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f3901ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f3901ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:57:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f3901ffb700 10 monclient: dump: 2023-09-30T01:57:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:25.777 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:25.778 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f390933b700 10 monclient: shutdown 2023-09-30T01:57:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f390933b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f390933b700 1 -- 172.21.15.7:0/2897866567 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39040a2390 msgr2=0x7f3904096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f390933b700 1 --2- 172.21.15.7:0/2897866567 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39040a2390 0x7f3904096e20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f38ec0099b0 tx=0x7f38ec009250 comp rx=0 tx=0).stop 2023-09-30T01:57:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f390933b700 1 -- 172.21.15.7:0/2897866567 shutdown_connections 2023-09-30T01:57:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f390933b700 1 --2- 172.21.15.7:0/2897866567 >> v2:172.21.15.50:3300/0 conn(0x7f3904097360 0x7f3904139720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f390933b700 1 --2- 172.21.15.7:0/2897866567 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39040a2390 0x7f3904096e20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:25.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.767+0000 7f390933b700 1 -- 172.21.15.7:0/2897866567 >> 172.21.15.7:0/2897866567 conn(0x7f3904070cf0 msgr2=0x7f39040710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:25.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.768+0000 7f390933b700 1 -- 172.21.15.7:0/2897866567 shutdown_connections 2023-09-30T01:57:25.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.768+0000 7f390933b700 1 -- 172.21.15.7:0/2897866567 wait complete. 2023-09-30T01:57:25.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.768+0000 7f390933b700 10 monclient: build_initial_monmap 2023-09-30T01:57:25.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.768+0000 7f390933b700 10 monclient: monmap: 2023-09-30T01:57:25.779 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:25.779 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:57:25.779 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:57:25.779 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:57:25.779 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:25.779 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:25.779 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:57:25.779 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:57:25.780 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:57:25.780 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:25.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.768+0000 7f390933b700 1 Processor -- start 2023-09-30T01:57:25.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f390933b700 1 -- start start 2023-09-30T01:57:25.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f390933b700 10 monclient: init 2023-09-30T01:57:25.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f390933b700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:25.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f390933b700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:57:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f390933b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3904096a50 0x7f390409bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f390933b700 10 monclient(hunting): picked mon.noname-b con 0x7f3904096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:57:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f390933b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3904097360 0x7f390409c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f390933b700 10 monclient(hunting): picked mon.noname-a con 0x7f3904097360 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:57:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f390933b700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39040a2390 0x7f390409c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f390933b700 10 monclient(hunting): picked mon.noname-c con 0x7f39040a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f390933b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f390933b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390409f360 con 0x7f3904097360 2023-09-30T01:57:25.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f390933b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:25.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f390933b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39040a02a0 con 0x7f3904096a50 2023-09-30T01:57:25.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f390933b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:25.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f390933b700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39041334e0 con 0x7f39040a2390 2023-09-30T01:57:25.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f390933b700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:25.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f390933b700 10 monclient(hunting): authenticate will time out at 1334.834509s 2023-09-30T01:57:25.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39037fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39040a2390 0x7f390409c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:25.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39037fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39040a2390 0x7f390409c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38520/0 (socket says 172.21.15.7:38520) 2023-09-30T01:57:25.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39027fc700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3904097360 0x7f390409c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:25.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39037fe700 1 -- 172.21.15.7:0/3799830324 learned_addr learned my addr 172.21.15.7:0/3799830324 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39037fe700 10 monclient(hunting): get_auth_request con 0x7f39040a2390 auth_method 0 2023-09-30T01:57:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39037fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39037fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39037fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39027fc700 10 monclient(hunting): get_auth_request con 0x7f3904097360 auth_method 0 2023-09-30T01:57:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39027fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39027fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39027fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39037fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39037fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39037fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39027fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:25.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39027fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:25.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39027fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:25.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39037fe700 10 monclient(hunting): handle_auth_done global_id 4538 payload 824 2023-09-30T01:57:25.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f3902ffd700 1 --2- 172.21.15.7:0/3799830324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3904096a50 0x7f390409bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:25.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39037fe700 1 -- 172.21.15.7:0/3799830324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3904096a50 msgr2=0x7f390409bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:25.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39037fe700 1 --2- 172.21.15.7:0/3799830324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3904096a50 0x7f390409bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:25.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39037fe700 1 -- 172.21.15.7:0/3799830324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3904097360 msgr2=0x7f390409c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:25.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39037fe700 1 --2- 172.21.15.7:0/3799830324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3904097360 0x7f390409c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:25.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39037fe700 10 monclient: _finish_hunting 0 2023-09-30T01:57:25.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39037fe700 1 monclient: found mon.noname-c 2023-09-30T01:57:25.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39037fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:25.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39037fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:57:25.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39037fe700 1 -- 172.21.15.7:0/3799830324 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ec0094f0 con 0x7f39040a2390 2023-09-30T01:57:25.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f3902ffd700 10 monclient: get_auth_request con 0x7f3904096a50 auth_method 0 2023-09-30T01:57:25.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f39027fc700 1 --2- 172.21.15.7:0/3799830324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3904097360 0x7f390409c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:25.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.769+0000 7f3902ffd700 1 --2- 172.21.15.7:0/3799830324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3904096a50 0x7f390409bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:57:25.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f39037fe700 1 -- 172.21.15.7:0/3799830324 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39040a2390 msgr2=0x7f390409c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:25.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f39037fe700 1 --2- 172.21.15.7:0/3799830324 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39040a2390 0x7f390409c680 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009e90 tx=0x7f38fc00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:25.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f38faffd700 1 -- 172.21.15.7:0/3799830324 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc00cd60 con 0x7f39040a2390 2023-09-30T01:57:25.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f38faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:25.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f38faffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:57:25.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f38faffd700 10 monclient: dump: 2023-09-30T01:57:25.789 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:25.789 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:25.789 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:25.790 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f38faffd700 10 monclient: _finish_auth 0 2023-09-30T01:57:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f38faffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f390933b700 5 monclient: authenticate success, global_id 4538 2023-09-30T01:57:25.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f38faffd700 1 -- 172.21.15.7:0/3799830324 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38fc00f040 con 0x7f39040a2390 2023-09-30T01:57:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f38faffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f38faffd700 1 -- 172.21.15.7:0/3799830324 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc0146d0 con 0x7f39040a2390 2023-09-30T01:57:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f390933b700 10 monclient: _renew_subs 2023-09-30T01:57:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f390933b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f390933b700 1 -- 172.21.15.7:0/3799830324 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390409f080 con 0x7f39040a2390 2023-09-30T01:57:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f39017fa700 4 set_mon_vals no callback set 2023-09-30T01:57:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f38faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:25.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f38faffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:57:25.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f38faffd700 10 monclient: dump: 2023-09-30T01:57:25.792 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:25.792 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:25.792 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:25.792 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:25.792 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:25.792 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:25.792 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:25.792 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:25.792 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:25.792 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:25.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f390933b700 10 monclient: _renew_subs 2023-09-30T01:57:25.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f390933b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:25.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.772+0000 7f390933b700 1 -- 172.21.15.7:0/3799830324 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390409ed70 con 0x7f39040a2390 2023-09-30T01:57:25.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.773+0000 7f38faffd700 1 -- 172.21.15.7:0/3799830324 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f38fc014a80 con 0x7f39040a2390 2023-09-30T01:57:25.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.773+0000 7f390933b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:25.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.773+0000 7f390933b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:25.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.773+0000 7f390933b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:25.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.773+0000 7f390933b700 1 -- 172.21.15.7:0/3799830324 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39040a4ba0 con 0x7f39040a2390 2023-09-30T01:57:25.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.773+0000 7f38faffd700 1 --2- 172.21.15.7:0/3799830324 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f38e40357e0 0x7f38e4037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:25.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.773+0000 7f38faffd700 1 -- 172.21.15.7:0/3799830324 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f38fc048720 con 0x7f39040a2390 2023-09-30T01:57:25.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.773+0000 7f3902ffd700 1 --2- 172.21.15.7:0/3799830324 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f38e40357e0 0x7f38e4037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:25.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.773+0000 7f3902ffd700 10 monclient: get_auth_request con 0x7f38e40357e0 auth_method 0 2023-09-30T01:57:25.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.776+0000 7f3902ffd700 1 --2- 172.21.15.7:0/3799830324 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f38e40357e0 0x7f38e4037c90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f38f400a900 tx=0x7f38f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:25.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.778+0000 7f38faffd700 1 -- 172.21.15.7:0/3799830324 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f38fc01ad20 con 0x7f39040a2390 2023-09-30T01:57:25.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.778+0000 7f38faffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:25.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:25.778+0000 7f38faffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:57:26.011 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:26.011+0000 7f390933b700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:57:26.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:26.011+0000 7f390933b700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:26.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:26.011+0000 7f390933b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:26.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:26.011+0000 7f390933b700 1 -- 172.21.15.7:0/3799830324 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3904064c10 con 0x7f39040a2390 2023-09-30T01:57:26.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:26.012+0000 7f38faffd700 1 -- 172.21.15.7:0/3799830324 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f38fc018080 con 0x7f39040a2390 2023-09-30T01:57:26.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:26.012+0000 7f38faffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:26.012 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:26.012+0000 7f38faffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:57:26.013 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:57:26.013 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:57:26.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:26.014+0000 7f390933b700 1 -- 172.21.15.7:0/3799830324 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f38e40357e0 msgr2=0x7f38e4037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:26.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:26.014+0000 7f390933b700 1 --2- 172.21.15.7:0/3799830324 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f38e40357e0 0x7f38e4037c90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f38f400a900 tx=0x7f38f4008040 comp rx=0 tx=0).stop 2023-09-30T01:57:26.014 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:26.014+0000 7f390933b700 10 monclient: shutdown 2023-09-30T01:57:26.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:26.014+0000 7f390933b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:26.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:26.014+0000 7f390933b700 1 -- 172.21.15.7:0/3799830324 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39040a2390 msgr2=0x7f390409c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:26.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:26.014+0000 7f390933b700 1 --2- 172.21.15.7:0/3799830324 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39040a2390 0x7f390409c680 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009e90 tx=0x7f38fc00b040 comp rx=0 tx=0).stop 2023-09-30T01:57:26.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:26.014+0000 7f390933b700 1 -- 172.21.15.7:0/3799830324 shutdown_connections 2023-09-30T01:57:26.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:26.014+0000 7f390933b700 1 --2- 172.21.15.7:0/3799830324 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f38e40357e0 0x7f38e4037c90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:26.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:26.014+0000 7f390933b700 1 --2- 172.21.15.7:0/3799830324 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f39040a2390 0x7f390409c680 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:26.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:26.014+0000 7f390933b700 1 --2- 172.21.15.7:0/3799830324 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3904096a50 0x7f390409bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:26.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:26.014+0000 7f390933b700 1 --2- 172.21.15.7:0/3799830324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3904097360 0x7f390409c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:26.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:26.014+0000 7f390933b700 1 -- 172.21.15.7:0/3799830324 >> 172.21.15.7:0/3799830324 conn(0x7f3904070cf0 msgr2=0x7f39040a3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:26.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:26.014+0000 7f390933b700 1 -- 172.21.15.7:0/3799830324 shutdown_connections 2023-09-30T01:57:26.015 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:26.014+0000 7f390933b700 1 -- 172.21.15.7:0/3799830324 wait complete. 2023-09-30T01:57:29.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:57:29.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.158+0000 7f889cdd9700 10 monclient: get_monmap_and_config 2023-09-30T01:57:29.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.158+0000 7f889cdd9700 10 monclient: build_initial_monmap 2023-09-30T01:57:29.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.158+0000 7f889cdd9700 10 monclient: monmap: 2023-09-30T01:57:29.159 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:29.159 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:29.159 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:57:29.159644+0000 2023-09-30T01:57:29.159 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:57:29.159644+0000 2023-09-30T01:57:29.159 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:29.159 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:29.159 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:57:29.159 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:57:29.159 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:57:29.159 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:57:29.159 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:57:29.160 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:29.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.159+0000 7f889cdd9700 1 Processor -- start 2023-09-30T01:57:29.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.159+0000 7f889cdd9700 1 -- start start 2023-09-30T01:57:29.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.159+0000 7f889cdd9700 10 monclient: init 2023-09-30T01:57:29.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.159+0000 7f889cdd9700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:29.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.159+0000 7f889cdd9700 10 monclient: _add_conns ranks=[0,4,2,1,3] 2023-09-30T01:57:29.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.159+0000 7f889cdd9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88980a29e0 addr v1:172.21.15.7:6789/0 2023-09-30T01:57:29.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.159+0000 7f889cdd9700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8898097160 0x7f8898137550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:29.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.159+0000 7f889cdd9700 10 monclient(hunting): picked mon.noname-c con 0x7f8898097160 addr v2:172.21.15.50:3300/0 2023-09-30T01:57:29.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.159+0000 7f889cdd9700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8898097560 0x7f8898139aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:29.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.159+0000 7f889cdd9700 10 monclient(hunting): picked mon.noname-b con 0x7f8898097560 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:29.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.159+0000 7f889cdd9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8898069cc0 con 0x7f88980a29e0 2023-09-30T01:57:29.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.159+0000 7f889cdd9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:29.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.159+0000 7f889cdd9700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8898098ea0 con 0x7f8898097160 2023-09-30T01:57:29.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.159+0000 7f889cdd9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:29.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.159+0000 7f889cdd9700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88980a2db0 con 0x7f8898097560 2023-09-30T01:57:29.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.159+0000 7f889cdd9700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:29.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.159+0000 7f889cdd9700 10 monclient(hunting): authenticate will time out at 1338.224923s 2023-09-30T01:57:29.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f889659c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f88980a29e0 0x7f8898096a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35098/0 (socket says 172.21.15.7:35098) 2023-09-30T01:57:29.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8896d9d700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8898097560 0x7f8898139aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:29.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f889659c700 1 -- 172.21.15.7:0/1909532509 learned_addr learned my addr 172.21.15.7:0/1909532509 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:57:29.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8896d9d700 10 monclient(hunting): get_auth_request con 0x7f8898097560 auth_method 0 2023-09-30T01:57:29.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8896d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:29.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8896d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:29.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8896d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:29.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8896d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:29.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8896d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:29.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8896d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:29.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8895d9b700 1 --2- 172.21.15.7:0/1909532509 >> v2:172.21.15.50:3300/0 conn(0x7f8898097160 0x7f8898137550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:29.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8895d9b700 10 monclient(hunting): get_auth_request con 0x7f8898097160 auth_method 0 2023-09-30T01:57:29.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8895d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:29.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8895d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:29.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8895d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:29.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8896d9d700 10 monclient(hunting): handle_auth_done global_id 4541 payload 274 2023-09-30T01:57:29.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8896d9d700 1 -- 172.21.15.7:0/1909532509 >> v2:172.21.15.50:3300/0 conn(0x7f8898097160 msgr2=0x7f8898137550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:29.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8896d9d700 1 --2- 172.21.15.7:0/1909532509 >> v2:172.21.15.50:3300/0 conn(0x7f8898097160 0x7f8898137550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:29.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8896d9d700 1 -- 172.21.15.7:0/1909532509 >> v1:172.21.15.7:6789/0 conn(0x7f88980a29e0 legacy=0x7f8898096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:29.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8896d9d700 10 monclient: _finish_hunting 0 2023-09-30T01:57:29.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8896d9d700 1 monclient: found mon.noname-b 2023-09-30T01:57:29.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8896d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:29.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8896d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:57:29.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.160+0000 7f8896d9d700 1 -- 172.21.15.7:0/1909532509 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8898098270 con 0x7f8898097560 2023-09-30T01:57:29.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.162+0000 7f8896d9d700 1 -- 172.21.15.7:0/1909532509 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8898097560 msgr2=0x7f8898139aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:29.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.162+0000 7f8896d9d700 1 --2- 172.21.15.7:0/1909532509 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8898097560 0x7f8898139aa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8890009e60 tx=0x7f8890009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4fbfb1da25a6c0b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:29.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.162+0000 7f889559a700 1 -- 172.21.15.7:0/1909532509 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889000e070 con 0x7f8898097560 2023-09-30T01:57:29.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.162+0000 7f889559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:29.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.162+0000 7f889559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:57:29.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.162+0000 7f889559a700 10 monclient: dump: 2023-09-30T01:57:29.170 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:29.170 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:29.171 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:29.171 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:29.171 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:29.171 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:29.171 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:29.171 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:29.171 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:29.171 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:29.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.162+0000 7f889559a700 10 monclient: _finish_auth 0 2023-09-30T01:57:29.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.162+0000 7f889559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:29.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.162+0000 7f889559a700 1 -- 172.21.15.7:0/1909532509 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889000be00 con 0x7f8898097560 2023-09-30T01:57:29.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.162+0000 7f889cdd9700 5 monclient: authenticate success, global_id 4541 2023-09-30T01:57:29.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.162+0000 7f889cdd9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:57:29.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.162+0000 7f889559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:29.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.162+0000 7f889cdd9700 10 monclient: get_monmap_and_config success 2023-09-30T01:57:29.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.162+0000 7f889559a700 1 -- 172.21.15.7:0/1909532509 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890012660 con 0x7f8898097560 2023-09-30T01:57:29.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.162+0000 7f889559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:29.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.162+0000 7f889cdd9700 4 set_mon_vals no callback set 2023-09-30T01:57:29.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.162+0000 7f889559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:57:29.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.162+0000 7f889559a700 10 monclient: dump: 2023-09-30T01:57:29.172 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:29.172 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:29.172 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:29.172 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:29.173 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:29.173 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:29.173 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:29.173 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:29.173 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:29.173 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:29.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.163+0000 7f889cdd9700 10 monclient: shutdown 2023-09-30T01:57:29.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.163+0000 7f889cdd9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:29.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.163+0000 7f889cdd9700 1 -- 172.21.15.7:0/1909532509 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8898097560 msgr2=0x7f8898139aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:29.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.163+0000 7f889cdd9700 1 --2- 172.21.15.7:0/1909532509 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8898097560 0x7f8898139aa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8890009e60 tx=0x7f8890009780 comp rx=0 tx=0).stop 2023-09-30T01:57:29.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.163+0000 7f889cdd9700 1 -- 172.21.15.7:0/1909532509 shutdown_connections 2023-09-30T01:57:29.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.163+0000 7f889cdd9700 1 --2- 172.21.15.7:0/1909532509 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8898097560 0x7f8898139aa0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.163+0000 7f889cdd9700 1 --2- 172.21.15.7:0/1909532509 >> v2:172.21.15.50:3300/0 conn(0x7f8898097160 0x7f8898137550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.163+0000 7f889cdd9700 1 -- 172.21.15.7:0/1909532509 >> 172.21.15.7:0/1909532509 conn(0x7f88980710d0 msgr2=0x7f88980714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.163+0000 7f889cdd9700 1 -- 172.21.15.7:0/1909532509 shutdown_connections 2023-09-30T01:57:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.163+0000 7f889cdd9700 1 -- 172.21.15.7:0/1909532509 wait complete. 2023-09-30T01:57:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.163+0000 7f889cdd9700 10 monclient: build_initial_monmap 2023-09-30T01:57:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889cdd9700 10 monclient: monmap: 2023-09-30T01:57:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:57:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:57:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:57:29.174 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:29.175 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:29.175 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:57:29.175 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:57:29.175 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:57:29.175 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:29.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889cdd9700 1 Processor -- start 2023-09-30T01:57:29.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889cdd9700 1 -- start start 2023-09-30T01:57:29.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889cdd9700 10 monclient: init 2023-09-30T01:57:29.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889cdd9700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:29.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889cdd9700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:57:29.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889cdd9700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8898097160 0x7f88981300b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:29.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889cdd9700 10 monclient(hunting): picked mon.noname-c con 0x7f8898097160 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:29.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889cdd9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8898097560 0x7f889809ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:29.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889cdd9700 10 monclient(hunting): picked mon.noname-a con 0x7f8898097560 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:57:29.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889cdd9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88980a29e0 0x7f889809f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:29.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889cdd9700 10 monclient(hunting): picked mon.noname-b con 0x7f88980a29e0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:57:29.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889cdd9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:29.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889cdd9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8898135140 con 0x7f8898097560 2023-09-30T01:57:29.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889cdd9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:29.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889cdd9700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889811d4b0 con 0x7f88980a29e0 2023-09-30T01:57:29.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889cdd9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:29.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889cdd9700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88980a0a80 con 0x7f8898097160 2023-09-30T01:57:29.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889cdd9700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:29.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889cdd9700 10 monclient(hunting): authenticate will time out at 1338.229836s 2023-09-30T01:57:29.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889659c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8898097160 0x7f88981300b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:29.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889659c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8898097160 0x7f88981300b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:38546/0 (socket says 172.21.15.7:38546) 2023-09-30T01:57:29.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889659c700 1 -- 172.21.15.7:0/319506843 learned_addr learned my addr 172.21.15.7:0/319506843 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:29.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889659c700 10 monclient(hunting): get_auth_request con 0x7f8898097160 auth_method 0 2023-09-30T01:57:29.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:29.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:29.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f889659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:29.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.164+0000 7f8895d9b700 1 --2- 172.21.15.7:0/319506843 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8898097560 0x7f889809ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:29.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f8895d9b700 10 monclient(hunting): get_auth_request con 0x7f8898097560 auth_method 0 2023-09-30T01:57:29.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f8895d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:29.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f8895d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:29.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f8895d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:29.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f889659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:29.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f889659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:29.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f889659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:29.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f8896d9d700 1 --2- 172.21.15.7:0/319506843 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88980a29e0 0x7f889809f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:29.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f8896d9d700 10 monclient(hunting): get_auth_request con 0x7f88980a29e0 auth_method 0 2023-09-30T01:57:29.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f8896d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:29.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f8896d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:29.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f8896d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:29.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f8895d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:29.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f8895d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:29.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f8895d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:29.179 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f8896d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:29.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f8896d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:29.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f8896d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:29.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f889659c700 10 monclient(hunting): handle_auth_done global_id 4544 payload 824 2023-09-30T01:57:29.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f889659c700 1 -- 172.21.15.7:0/319506843 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88980a29e0 msgr2=0x7f889809f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:29.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f889659c700 1 --2- 172.21.15.7:0/319506843 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88980a29e0 0x7f889809f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:29.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f889659c700 1 -- 172.21.15.7:0/319506843 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8898097560 msgr2=0x7f889809ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:29.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f889659c700 1 --2- 172.21.15.7:0/319506843 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8898097560 0x7f889809ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:29.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f889659c700 10 monclient: _finish_hunting 0 2023-09-30T01:57:29.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f889659c700 1 monclient: found mon.noname-c 2023-09-30T01:57:29.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f889659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:29.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f889659c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:57:29.180 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.165+0000 7f889659c700 1 -- 172.21.15.7:0/319506843 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8890009a20 con 0x7f8898097160 2023-09-30T01:57:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f889659c700 1 -- 172.21.15.7:0/319506843 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8898097160 msgr2=0x7f88981300b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f889659c700 1 --2- 172.21.15.7:0/319506843 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8898097160 0x7f88981300b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f888800c5d0 tx=0x7f8888007f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f88867fc700 1 -- 172.21.15.7:0/319506843 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8888007530 con 0x7f8898097160 2023-09-30T01:57:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f88867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f88867fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:57:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f88867fc700 10 monclient: dump: 2023-09-30T01:57:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:29.183 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:29.184 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:29.184 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:29.184 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:29.184 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:29.184 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:29.184 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:29.184 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:29.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f88867fc700 10 monclient: _finish_auth 0 2023-09-30T01:57:29.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f88867fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:29.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f889cdd9700 5 monclient: authenticate success, global_id 4544 2023-09-30T01:57:29.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f889cdd9700 10 monclient: _renew_subs 2023-09-30T01:57:29.184 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f88867fc700 1 -- 172.21.15.7:0/319506843 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88880076b0 con 0x7f8898097160 2023-09-30T01:57:29.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f889cdd9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:29.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f889cdd9700 1 -- 172.21.15.7:0/319506843 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8898142460 con 0x7f8898097160 2023-09-30T01:57:29.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f88867fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:29.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f88867fc700 1 -- 172.21.15.7:0/319506843 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8888005a40 con 0x7f8898097160 2023-09-30T01:57:29.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f88867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:29.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f88867fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:57:29.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f8894d99700 4 set_mon_vals no callback set 2023-09-30T01:57:29.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f88867fc700 10 monclient: dump: 2023-09-30T01:57:29.185 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:29.185 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:29.185 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:29.186 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:29.186 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:29.186 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:29.186 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:29.186 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:29.186 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:29.186 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:29.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f889cdd9700 10 monclient: _renew_subs 2023-09-30T01:57:29.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f889cdd9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:29.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.167+0000 7f889cdd9700 1 -- 172.21.15.7:0/319506843 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8898136a40 con 0x7f8898097160 2023-09-30T01:57:29.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.168+0000 7f88867fc700 1 -- 172.21.15.7:0/319506843 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f8888014420 con 0x7f8898097160 2023-09-30T01:57:29.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.168+0000 7f889cdd9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:29.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.168+0000 7f889cdd9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:29.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.168+0000 7f889cdd9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:29.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.168+0000 7f889cdd9700 1 -- 172.21.15.7:0/319506843 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f889811d2e0 con 0x7f8898097160 2023-09-30T01:57:29.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.168+0000 7f88867fc700 1 --2- 172.21.15.7:0/319506843 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f88780357e0 0x7f8878037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:29.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.168+0000 7f88867fc700 1 -- 172.21.15.7:0/319506843 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f8888047a80 con 0x7f8898097160 2023-09-30T01:57:29.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.169+0000 7f8895d9b700 1 --2- 172.21.15.7:0/319506843 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f88780357e0 0x7f8878037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:29.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.169+0000 7f8895d9b700 10 monclient: get_auth_request con 0x7f88780357e0 auth_method 0 2023-09-30T01:57:29.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.171+0000 7f8895d9b700 1 --2- 172.21.15.7:0/319506843 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f88780357e0 0x7f8878037c90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f888c006fd0 tx=0x7f888c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:29.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.173+0000 7f88867fc700 1 -- 172.21.15.7:0/319506843 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8888005df0 con 0x7f8898097160 2023-09-30T01:57:29.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.173+0000 7f88867fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:29.187 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.173+0000 7f88867fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:57:29.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.412+0000 7f889cdd9700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:57:29.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.412+0000 7f889cdd9700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:29.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.412+0000 7f889cdd9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:29.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.412+0000 7f889cdd9700 1 -- 172.21.15.7:0/319506843 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8898064c10 con 0x7f8898097160 2023-09-30T01:57:29.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.412+0000 7f88867fc700 1 -- 172.21.15.7:0/319506843 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f88880102c0 con 0x7f8898097160 2023-09-30T01:57:29.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.412+0000 7f88867fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:29.412 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.412+0000 7f88867fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:57:29.413 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:57:29.413 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:57:29.414 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.414+0000 7f889cdd9700 1 -- 172.21.15.7:0/319506843 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f88780357e0 msgr2=0x7f8878037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:29.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.414+0000 7f889cdd9700 1 --2- 172.21.15.7:0/319506843 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f88780357e0 0x7f8878037c90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f888c006fd0 tx=0x7f888c008040 comp rx=0 tx=0).stop 2023-09-30T01:57:29.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.414+0000 7f889cdd9700 10 monclient: shutdown 2023-09-30T01:57:29.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.414+0000 7f889cdd9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:29.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.414+0000 7f889cdd9700 1 -- 172.21.15.7:0/319506843 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8898097160 msgr2=0x7f88981300b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:29.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.414+0000 7f889cdd9700 1 --2- 172.21.15.7:0/319506843 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8898097160 0x7f88981300b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f888800c5d0 tx=0x7f8888007f40 comp rx=0 tx=0).stop 2023-09-30T01:57:29.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.414+0000 7f889cdd9700 1 -- 172.21.15.7:0/319506843 shutdown_connections 2023-09-30T01:57:29.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.414+0000 7f889cdd9700 1 --2- 172.21.15.7:0/319506843 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f88780357e0 0x7f8878037c90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:29.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.414+0000 7f889cdd9700 1 --2- 172.21.15.7:0/319506843 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88980a29e0 0x7f889809f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:29.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.414+0000 7f889cdd9700 1 --2- 172.21.15.7:0/319506843 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8898097160 0x7f88981300b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:29.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.414+0000 7f889cdd9700 1 --2- 172.21.15.7:0/319506843 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8898097560 0x7f889809ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:29.415 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.414+0000 7f889cdd9700 1 -- 172.21.15.7:0/319506843 >> 172.21.15.7:0/319506843 conn(0x7f88980710d0 msgr2=0x7f8898135350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:29.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.414+0000 7f889cdd9700 1 -- 172.21.15.7:0/319506843 shutdown_connections 2023-09-30T01:57:29.416 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:29.414+0000 7f889cdd9700 1 -- 172.21.15.7:0/319506843 wait complete. 2023-09-30T01:57:32.427 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:57:32.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.557+0000 7f554623e700 10 monclient: get_monmap_and_config 2023-09-30T01:57:32.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.557+0000 7f554623e700 10 monclient: build_initial_monmap 2023-09-30T01:57:32.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.558+0000 7f554623e700 10 monclient: monmap: 2023-09-30T01:57:32.558 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:32.559 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:32.559 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:57:32.559052+0000 2023-09-30T01:57:32.559 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:57:32.559052+0000 2023-09-30T01:57:32.559 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:32.559 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:32.559 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:57:32.560 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:57:32.560 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:57:32.560 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:57:32.560 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:57:32.560 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:32.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.558+0000 7f554623e700 1 Processor -- start 2023-09-30T01:57:32.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.558+0000 7f554623e700 1 -- start start 2023-09-30T01:57:32.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.558+0000 7f554623e700 10 monclient: init 2023-09-30T01:57:32.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f554623e700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:32.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f554623e700 10 monclient: _add_conns ranks=[4,0,3,1,2] 2023-09-30T01:57:32.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f554623e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5540096a50 0x7f55401327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:32.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f554623e700 10 monclient(hunting): picked mon.noname-c con 0x7f5540096a50 addr v2:172.21.15.50:3300/0 2023-09-30T01:57:32.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f554623e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f55400a2390 addr v1:172.21.15.7:6789/0 2023-09-30T01:57:32.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f554623e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5540097530 0x7f5540139720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:32.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f554623e700 10 monclient(hunting): picked mon.noname-a con 0x7f5540097530 addr v2:172.21.15.7:3300/0 2023-09-30T01:57:32.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f554623e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55400980e0 con 0x7f55400a2390 2023-09-30T01:57:32.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f554623e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:32.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f554623e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55400991a0 con 0x7f5540097530 2023-09-30T01:57:32.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f554623e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:32.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f554623e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5540099480 con 0x7f5540096a50 2023-09-30T01:57:32.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f554623e700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:32.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f554623e700 10 monclient(hunting): authenticate will time out at 1341.624275s 2023-09-30T01:57:32.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5540097530 0x7f5540139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:32.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5540097530 0x7f5540139720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53240/0 (socket says 172.21.15.7:53240) 2023-09-30T01:57:32.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553effd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f55400a2390 0x7f5540096e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35114/0 (socket says 172.21.15.7:35114) 2023-09-30T01:57:32.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553ffff700 1 -- 172.21.15.7:0/4011675134 learned_addr learned my addr 172.21.15.7:0/4011675134 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:32.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553ffff700 10 monclient(hunting): get_auth_request con 0x7f5540097530 auth_method 0 2023-09-30T01:57:32.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:32.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:32.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:32.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553f7fe700 1 --2- 172.21.15.7:0/4011675134 >> v2:172.21.15.50:3300/0 conn(0x7f5540096a50 0x7f55401327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:32.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:32.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:32.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:32.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553f7fe700 10 monclient(hunting): get_auth_request con 0x7f5540096a50 auth_method 0 2023-09-30T01:57:32.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:32.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:32.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:32.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:32.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:32.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:32.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553ffff700 10 monclient(hunting): handle_auth_done global_id 4524 payload 274 2023-09-30T01:57:32.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553ffff700 1 -- 172.21.15.7:0/4011675134 >> v2:172.21.15.50:3300/0 conn(0x7f5540096a50 msgr2=0x7f55401327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:32.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553ffff700 1 --2- 172.21.15.7:0/4011675134 >> v2:172.21.15.50:3300/0 conn(0x7f5540096a50 0x7f55401327a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:32.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553ffff700 1 -- 172.21.15.7:0/4011675134 >> v1:172.21.15.7:6789/0 conn(0x7f55400a2390 legacy=0x7f5540096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:32.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553ffff700 10 monclient: _finish_hunting 0 2023-09-30T01:57:32.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553ffff700 1 monclient: found mon.noname-a 2023-09-30T01:57:32.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:32.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:57:32.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.559+0000 7f553ffff700 1 -- 172.21.15.7:0/4011675134 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5540097bf0 con 0x7f5540097530 2023-09-30T01:57:32.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f553ffff700 1 -- 172.21.15.7:0/4011675134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5540097530 msgr2=0x7f5540139720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:32.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f553ffff700 1 --2- 172.21.15.7:0/4011675134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5540097530 0x7f5540139720 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f55400a2b00 tx=0x7f5538009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff23e803201eb35c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:32.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f553e7fc700 1 -- 172.21.15.7:0/4011675134 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553800e070 con 0x7f5540097530 2023-09-30T01:57:32.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f553e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:32.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f553e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:57:32.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f553e7fc700 10 monclient: dump: 2023-09-30T01:57:32.582 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:32.582 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:32.582 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:32.582 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:32.583 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:32.583 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:32.583 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:32.583 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:32.583 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:32.583 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:32.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f553e7fc700 10 monclient: _finish_auth 0 2023-09-30T01:57:32.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f553e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:32.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f553e7fc700 1 -- 172.21.15.7:0/4011675134 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553800bbf0 con 0x7f5540097530 2023-09-30T01:57:32.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f554623e700 5 monclient: authenticate success, global_id 4524 2023-09-30T01:57:32.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f554623e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:57:32.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f553e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:32.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f554623e700 10 monclient: get_monmap_and_config success 2023-09-30T01:57:32.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f553e7fc700 1 -- 172.21.15.7:0/4011675134 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55380092e0 con 0x7f5540097530 2023-09-30T01:57:32.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f553e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:32.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f554623e700 4 set_mon_vals no callback set 2023-09-30T01:57:32.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f553e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:57:32.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f553e7fc700 10 monclient: dump: 2023-09-30T01:57:32.584 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:32.585 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:32.585 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:32.585 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:32.585 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:32.585 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:32.585 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:32.585 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:32.585 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:32.585 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:32.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f554623e700 10 monclient: shutdown 2023-09-30T01:57:32.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f554623e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:32.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f554623e700 1 -- 172.21.15.7:0/4011675134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5540097530 msgr2=0x7f5540139720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:32.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f554623e700 1 --2- 172.21.15.7:0/4011675134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5540097530 0x7f5540139720 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f55400a2b00 tx=0x7f5538009840 comp rx=0 tx=0).stop 2023-09-30T01:57:32.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f554623e700 1 -- 172.21.15.7:0/4011675134 shutdown_connections 2023-09-30T01:57:32.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f554623e700 1 --2- 172.21.15.7:0/4011675134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5540097530 0x7f5540139720 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:32.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f554623e700 1 --2- 172.21.15.7:0/4011675134 >> v2:172.21.15.50:3300/0 conn(0x7f5540096a50 0x7f55401327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:32.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f554623e700 1 -- 172.21.15.7:0/4011675134 >> 172.21.15.7:0/4011675134 conn(0x7f5540070cf0 msgr2=0x7f55400710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:32.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f554623e700 1 -- 172.21.15.7:0/4011675134 shutdown_connections 2023-09-30T01:57:32.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.562+0000 7f554623e700 1 -- 172.21.15.7:0/4011675134 wait complete. 2023-09-30T01:57:32.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.563+0000 7f554623e700 10 monclient: build_initial_monmap 2023-09-30T01:57:32.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.563+0000 7f554623e700 10 monclient: monmap: 2023-09-30T01:57:32.589 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:32.589 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:57:32.590 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:57:32.590 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:57:32.590 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:32.590 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:32.590 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:57:32.590 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:57:32.590 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:57:32.590 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:32.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.563+0000 7f554623e700 1 Processor -- start 2023-09-30T01:57:32.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.563+0000 7f554623e700 1 -- start start 2023-09-30T01:57:32.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.563+0000 7f554623e700 10 monclient: init 2023-09-30T01:57:32.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f554623e700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:32.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f554623e700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:57:32.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f554623e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5540096a50 0x7f554009ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:32.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f554623e700 10 monclient(hunting): picked mon.noname-a con 0x7f5540096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:57:32.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f554623e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5540097530 0x7f554009bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:32.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f554623e700 10 monclient(hunting): picked mon.noname-b con 0x7f5540097530 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:57:32.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f554623e700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f55400a2390 0x7f554009c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:32.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f554623e700 10 monclient(hunting): picked mon.noname-c con 0x7f55400a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:32.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f554623e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:32.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f554623e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55401334e0 con 0x7f5540096a50 2023-09-30T01:57:32.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f554623e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:32.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f554623e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55400a15f0 con 0x7f5540097530 2023-09-30T01:57:32.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f554623e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:32.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f554623e700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55401303e0 con 0x7f55400a2390 2023-09-30T01:57:32.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f554623e700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:32.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f554623e700 10 monclient(hunting): authenticate will time out at 1341.629424s 2023-09-30T01:57:32.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5540096a50 0x7f554009ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:32.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553ffff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f55400a2390 0x7f554009c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:32.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5540096a50 0x7f554009ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53246/0 (socket says 172.21.15.7:53246) 2023-09-30T01:57:32.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553f7fe700 1 -- 172.21.15.7:0/264206028 learned_addr learned my addr 172.21.15.7:0/264206028 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:32.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553f7fe700 10 monclient(hunting): get_auth_request con 0x7f5540096a50 auth_method 0 2023-09-30T01:57:32.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:32.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:32.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:32.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553ffff700 10 monclient(hunting): get_auth_request con 0x7f55400a2390 auth_method 0 2023-09-30T01:57:32.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:32.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:32.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:32.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:32.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:32.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:32.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:32.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:32.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:32.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553effd700 1 --2- 172.21.15.7:0/264206028 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5540097530 0x7f554009bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:32.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553effd700 10 monclient(hunting): get_auth_request con 0x7f5540097530 auth_method 0 2023-09-30T01:57:32.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:32.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:32.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:32.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553f7fe700 10 monclient(hunting): handle_auth_done global_id 4527 payload 824 2023-09-30T01:57:32.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553f7fe700 1 -- 172.21.15.7:0/264206028 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f55400a2390 msgr2=0x7f554009c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:32.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553f7fe700 1 --2- 172.21.15.7:0/264206028 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f55400a2390 0x7f554009c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:32.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553f7fe700 1 -- 172.21.15.7:0/264206028 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5540097530 msgr2=0x7f554009bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:32.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553f7fe700 1 --2- 172.21.15.7:0/264206028 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5540097530 0x7f554009bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:32.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553f7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:57:32.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553f7fe700 1 monclient: found mon.noname-a 2023-09-30T01:57:32.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:32.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:57:32.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553f7fe700 1 -- 172.21.15.7:0/264206028 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5538009ae0 con 0x7f5540096a50 2023-09-30T01:57:32.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.564+0000 7f553effd700 1 --2- 172.21.15.7:0/264206028 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5540097530 0x7f554009bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:57:32.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f553f7fe700 1 -- 172.21.15.7:0/264206028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5540096a50 msgr2=0x7f554009ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:32.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f553f7fe700 1 --2- 172.21.15.7:0/264206028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5540096a50 0x7f554009ba50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f553000ecb0 tx=0x7f55300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:32.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f55277fe700 1 -- 172.21.15.7:0/264206028 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553000cad0 con 0x7f5540096a50 2023-09-30T01:57:32.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f55277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:32.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f55277fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:57:32.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f55277fe700 10 monclient: dump: 2023-09-30T01:57:32.595 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:32.595 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:32.596 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:32.596 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:32.596 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:32.596 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:32.596 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:32.596 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:32.596 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:32.596 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:32.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f55277fe700 10 monclient: _finish_auth 0 2023-09-30T01:57:32.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f55277fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:32.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f55277fe700 1 -- 172.21.15.7:0/264206028 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553000cc30 con 0x7f5540096a50 2023-09-30T01:57:32.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f554623e700 5 monclient: authenticate success, global_id 4527 2023-09-30T01:57:32.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f554623e700 10 monclient: _renew_subs 2023-09-30T01:57:32.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f554623e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:32.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f554623e700 1 -- 172.21.15.7:0/264206028 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f554009e070 con 0x7f5540096a50 2023-09-30T01:57:32.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f55277fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:32.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f55277fe700 1 -- 172.21.15.7:0/264206028 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5530013470 con 0x7f5540096a50 2023-09-30T01:57:32.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f55277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:32.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f55277fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:57:32.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f55277fe700 10 monclient: dump: 2023-09-30T01:57:32.597 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:32.597 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:32.597 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:32.598 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f554623e700 10 monclient: _renew_subs 2023-09-30T01:57:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f553dffb700 4 set_mon_vals no callback set 2023-09-30T01:57:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f554623e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.567+0000 7f554623e700 1 -- 172.21.15.7:0/264206028 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f554009d430 con 0x7f5540096a50 2023-09-30T01:57:32.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.568+0000 7f55277fe700 1 -- 172.21.15.7:0/264206028 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f5530013820 con 0x7f5540096a50 2023-09-30T01:57:32.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.568+0000 7f554623e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:32.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.568+0000 7f554623e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:32.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.568+0000 7f554623e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:32.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.568+0000 7f554623e700 1 -- 172.21.15.7:0/264206028 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5540071470 con 0x7f5540096a50 2023-09-30T01:57:32.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.568+0000 7f55277fe700 1 --2- 172.21.15.7:0/264206028 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f552c0357e0 0x7f552c037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:32.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.568+0000 7f55277fe700 1 -- 172.21.15.7:0/264206028 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f5530048060 con 0x7f5540096a50 2023-09-30T01:57:32.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.568+0000 7f553effd700 1 --2- 172.21.15.7:0/264206028 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f552c0357e0 0x7f552c037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:32.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.568+0000 7f553effd700 10 monclient: get_auth_request con 0x7f552c0357e0 auth_method 0 2023-09-30T01:57:32.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.571+0000 7f553effd700 1 --2- 172.21.15.7:0/264206028 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f552c0357e0 0x7f552c037c90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f554009d030 tx=0x7f5534009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:32.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.573+0000 7f55277fe700 1 -- 172.21.15.7:0/264206028 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f55300795a0 con 0x7f5540096a50 2023-09-30T01:57:32.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.573+0000 7f55277fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:32.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.573+0000 7f55277fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:57:32.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.807+0000 7f554623e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:57:32.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.807+0000 7f554623e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:32.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.807+0000 7f554623e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:32.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.807+0000 7f554623e700 1 -- 172.21.15.7:0/264206028 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5540064c10 con 0x7f5540096a50 2023-09-30T01:57:32.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.807+0000 7f55277fe700 1 -- 172.21.15.7:0/264206028 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f55300236b0 con 0x7f5540096a50 2023-09-30T01:57:32.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.807+0000 7f55277fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:32.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.807+0000 7f55277fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:57:32.808 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:57:32.808 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:57:32.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.809+0000 7f554623e700 1 -- 172.21.15.7:0/264206028 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f552c0357e0 msgr2=0x7f552c037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:32.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.809+0000 7f554623e700 1 --2- 172.21.15.7:0/264206028 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f552c0357e0 0x7f552c037c90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f554009d030 tx=0x7f5534009040 comp rx=0 tx=0).stop 2023-09-30T01:57:32.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.809+0000 7f554623e700 10 monclient: shutdown 2023-09-30T01:57:32.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.809+0000 7f554623e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:32.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.809+0000 7f554623e700 1 -- 172.21.15.7:0/264206028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5540096a50 msgr2=0x7f554009ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:32.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.809+0000 7f554623e700 1 --2- 172.21.15.7:0/264206028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5540096a50 0x7f554009ba50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f553000ecb0 tx=0x7f55300091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:32.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.810+0000 7f554623e700 1 -- 172.21.15.7:0/264206028 shutdown_connections 2023-09-30T01:57:32.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.810+0000 7f554623e700 1 --2- 172.21.15.7:0/264206028 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f552c0357e0 0x7f552c037c90 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:32.810 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.810+0000 7f554623e700 1 --2- 172.21.15.7:0/264206028 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5540096a50 0x7f554009ba50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:32.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.810+0000 7f554623e700 1 --2- 172.21.15.7:0/264206028 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f55400a2390 0x7f554009c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:32.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.810+0000 7f554623e700 1 --2- 172.21.15.7:0/264206028 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5540097530 0x7f554009bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:32.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.810+0000 7f554623e700 1 -- 172.21.15.7:0/264206028 >> 172.21.15.7:0/264206028 conn(0x7f5540070cf0 msgr2=0x7f55400a44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:32.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.810+0000 7f554623e700 1 -- 172.21.15.7:0/264206028 shutdown_connections 2023-09-30T01:57:32.811 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:32.810+0000 7f554623e700 1 -- 172.21.15.7:0/264206028 wait complete. 2023-09-30T01:57:35.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:57:35.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.953+0000 7f37bc738700 10 monclient: get_monmap_and_config 2023-09-30T01:57:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.953+0000 7f37bc738700 10 monclient: build_initial_monmap 2023-09-30T01:57:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.953+0000 7f37bc738700 10 monclient: monmap: 2023-09-30T01:57:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:57:35.954692+0000 2023-09-30T01:57:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:57:35.954692+0000 2023-09-30T01:57:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:35.954 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:57:35.955 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:57:35.955 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:57:35.955 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:57:35.955 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:57:35.955 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:35.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.954+0000 7f37bc738700 1 Processor -- start 2023-09-30T01:57:35.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.954+0000 7f37bc738700 1 -- start start 2023-09-30T01:57:35.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.954+0000 7f37bc738700 10 monclient: init 2023-09-30T01:57:35.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.954+0000 7f37bc738700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:35.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.954+0000 7f37bc738700 10 monclient: _add_conns ranks=[1,2,3,0,4] 2023-09-30T01:57:35.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.954+0000 7f37bc738700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f37b40a29e0 addr v1:172.21.15.50:6789/0 2023-09-30T01:57:35.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.954+0000 7f37bc738700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37b4097160 0x7f37b4137550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:35.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.954+0000 7f37bc738700 10 monclient(hunting): picked mon.noname-b con 0x7f37b4097160 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:35.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.954+0000 7f37bc738700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f37b4097530 0x7f37b4139aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:35.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.954+0000 7f37bc738700 10 monclient(hunting): picked mon.noname-a con 0x7f37b4097530 addr v2:172.21.15.7:3300/0 2023-09-30T01:57:35.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.954+0000 7f37bc738700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b4069cc0 con 0x7f37b40a29e0 2023-09-30T01:57:35.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.954+0000 7f37bc738700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.954+0000 7f37bc738700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37b4098ea0 con 0x7f37b4097530 2023-09-30T01:57:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.954+0000 7f37bc738700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.954+0000 7f37bc738700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b40a2db0 con 0x7f37b4097160 2023-09-30T01:57:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.954+0000 7f37bc738700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.954+0000 7f37bc738700 10 monclient(hunting): authenticate will time out at 1345.019934s 2023-09-30T01:57:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37bacd5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f37b4097530 0x7f37b4139aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37b9cd3700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37b4097160 0x7f37b4137550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37bacd5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f37b4097530 0x7f37b4139aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49958/0 (socket says 172.21.15.7:49958) 2023-09-30T01:57:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37bacd5700 1 -- 172.21.15.7:0/2952621586 learned_addr learned my addr 172.21.15.7:0/2952621586 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37bacd5700 10 monclient(hunting): get_auth_request con 0x7f37b4097530 auth_method 0 2023-09-30T01:57:35.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37bacd5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37bacd5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37bacd5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37b9cd3700 10 monclient(hunting): get_auth_request con 0x7f37b4097160 auth_method 0 2023-09-30T01:57:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37b9cd3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37b9cd3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37b9cd3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37bacd5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37bacd5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37bacd5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37b9cd3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37b9cd3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:35.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37b9cd3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37bacd5700 10 monclient(hunting): handle_auth_done global_id 4530 payload 274 2023-09-30T01:57:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37bacd5700 1 -- 172.21.15.7:0/2952621586 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37b4097160 msgr2=0x7f37b4137550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37bacd5700 1 --2- 172.21.15.7:0/2952621586 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37b4097160 0x7f37b4137550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37bacd5700 1 -- 172.21.15.7:0/2952621586 >> v1:172.21.15.50:6789/0 conn(0x7f37b40a29e0 legacy=0x7f37b4096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37bacd5700 10 monclient: _finish_hunting 0 2023-09-30T01:57:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37bacd5700 1 monclient: found mon.noname-a 2023-09-30T01:57:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37bacd5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37bacd5700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:57:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37bacd5700 1 -- 172.21.15.7:0/2952621586 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b4098270 con 0x7f37b4097530 2023-09-30T01:57:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.955+0000 7f37b9cd3700 1 --2- 172.21.15.7:0/2952621586 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37b4097160 0x7f37b4137550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:35.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.957+0000 7f37bacd5700 1 -- 172.21.15.7:0/2952621586 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37b4097530 msgr2=0x7f37b4139aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.957+0000 7f37bacd5700 1 --2- 172.21.15.7:0/2952621586 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37b4097530 0x7f37b4139aa0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f37ac009f20 tx=0x7f37ac009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdf82bdb08312211 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37b94d2700 1 -- 172.21.15.7:0/2952621586 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac00e070 con 0x7f37b4097530 2023-09-30T01:57:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37b94d2700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37b94d2700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:57:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37b94d2700 10 monclient: dump: 2023-09-30T01:57:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:35.966 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:35.967 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37b94d2700 10 monclient: _finish_auth 0 2023-09-30T01:57:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37b94d2700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37b94d2700 1 -- 172.21.15.7:0/2952621586 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ac00bba0 con 0x7f37b4097530 2023-09-30T01:57:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37bc738700 5 monclient: authenticate success, global_id 4530 2023-09-30T01:57:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37bc738700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:57:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37b94d2700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37b94d2700 1 -- 172.21.15.7:0/2952621586 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac0092e0 con 0x7f37b4097530 2023-09-30T01:57:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37b94d2700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:35.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37b94d2700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:57:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37b94d2700 10 monclient: dump: 2023-09-30T01:57:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:35.968 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:35.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37bc738700 10 monclient: get_monmap_and_config success 2023-09-30T01:57:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37bc738700 4 set_mon_vals no callback set 2023-09-30T01:57:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37bc738700 10 monclient: shutdown 2023-09-30T01:57:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37bc738700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37bc738700 1 -- 172.21.15.7:0/2952621586 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37b4097530 msgr2=0x7f37b4139aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37bc738700 1 --2- 172.21.15.7:0/2952621586 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37b4097530 0x7f37b4139aa0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f37ac009f20 tx=0x7f37ac009840 comp rx=0 tx=0).stop 2023-09-30T01:57:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37bc738700 1 -- 172.21.15.7:0/2952621586 shutdown_connections 2023-09-30T01:57:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37bc738700 1 --2- 172.21.15.7:0/2952621586 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37b4097530 0x7f37b4139aa0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37bc738700 1 --2- 172.21.15.7:0/2952621586 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37b4097160 0x7f37b4137550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37bc738700 1 -- 172.21.15.7:0/2952621586 >> 172.21.15.7:0/2952621586 conn(0x7f37b40710d0 msgr2=0x7f37b40714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37bc738700 1 -- 172.21.15.7:0/2952621586 shutdown_connections 2023-09-30T01:57:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.958+0000 7f37bc738700 1 -- 172.21.15.7:0/2952621586 wait complete. 2023-09-30T01:57:35.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.959+0000 7f37bc738700 10 monclient: build_initial_monmap 2023-09-30T01:57:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.959+0000 7f37bc738700 10 monclient: monmap: 2023-09-30T01:57:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:57:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:57:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:57:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:57:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:57:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:57:35.970 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:35.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.959+0000 7f37bc738700 1 Processor -- start 2023-09-30T01:57:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.959+0000 7f37bc738700 1 -- start start 2023-09-30T01:57:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.959+0000 7f37bc738700 10 monclient: init 2023-09-30T01:57:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bc738700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bc738700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:57:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bc738700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37b4097160 0x7f37b41415d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bc738700 10 monclient(hunting): picked mon.noname-a con 0x7f37b4097160 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:57:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bc738700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37b4097530 0x7f37b412e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bc738700 10 monclient(hunting): picked mon.noname-b con 0x7f37b4097530 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:57:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bc738700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37b40a29e0 0x7f37b409df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bc738700 10 monclient(hunting): picked mon.noname-c con 0x7f37b40a29e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bc738700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:35.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bc738700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b40a0ec0 con 0x7f37b4097160 2023-09-30T01:57:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bc738700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bc738700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b40a1220 con 0x7f37b4097530 2023-09-30T01:57:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bc738700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bc738700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b40a07d0 con 0x7f37b40a29e0 2023-09-30T01:57:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bc738700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bc738700 10 monclient(hunting): authenticate will time out at 1345.025165s 2023-09-30T01:57:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37ba4d4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37b4097160 0x7f37b41415d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37ba4d4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37b4097160 0x7f37b41415d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49960/0 (socket says 172.21.15.7:49960) 2023-09-30T01:57:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bacd5700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37b40a29e0 0x7f37b409df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37ba4d4700 1 -- 172.21.15.7:0/1545876537 learned_addr learned my addr 172.21.15.7:0/1545876537 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37ba4d4700 10 monclient(hunting): get_auth_request con 0x7f37b4097160 auth_method 0 2023-09-30T01:57:35.972 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37ba4d4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37ba4d4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37ba4d4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bacd5700 10 monclient(hunting): get_auth_request con 0x7f37b40a29e0 auth_method 0 2023-09-30T01:57:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bacd5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bacd5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bacd5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bacd5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bacd5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bacd5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37ba4d4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37ba4d4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:35.973 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37b9cd3700 1 --2- 172.21.15.7:0/1545876537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37b4097530 0x7f37b412e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37ba4d4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37b9cd3700 10 monclient(hunting): get_auth_request con 0x7f37b4097530 auth_method 0 2023-09-30T01:57:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37b9cd3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37b9cd3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37b9cd3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bacd5700 10 monclient(hunting): handle_auth_done global_id 4553 payload 824 2023-09-30T01:57:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bacd5700 1 -- 172.21.15.7:0/1545876537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37b4097530 msgr2=0x7f37b412e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bacd5700 1 --2- 172.21.15.7:0/1545876537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37b4097530 0x7f37b412e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bacd5700 1 -- 172.21.15.7:0/1545876537 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37b4097160 msgr2=0x7f37b41415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bacd5700 1 --2- 172.21.15.7:0/1545876537 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37b4097160 0x7f37b41415d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bacd5700 10 monclient: _finish_hunting 0 2023-09-30T01:57:35.974 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bacd5700 1 monclient: found mon.noname-c 2023-09-30T01:57:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bacd5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bacd5700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:57:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37bacd5700 1 -- 172.21.15.7:0/1545876537 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a4009420 con 0x7f37b40a29e0 2023-09-30T01:57:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37b9cd3700 1 --2- 172.21.15.7:0/1545876537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37b4097530 0x7f37b412e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:57:35.975 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.960+0000 7f37ba4d4700 1 --2- 172.21.15.7:0/1545876537 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37b4097160 0x7f37b41415d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.962+0000 7f37bacd5700 1 -- 172.21.15.7:0/1545876537 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37b40a29e0 msgr2=0x7f37b409df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:35.977 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.962+0000 7f37bacd5700 1 --2- 172.21.15.7:0/1545876537 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37b40a29e0 0x7f37b409df20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f37ac014000 tx=0x7f37ac017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.962+0000 7f37aa7fc700 1 -- 172.21.15.7:0/1545876537 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac0071b0 con 0x7f37b40a29e0 2023-09-30T01:57:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.962+0000 7f37aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.962+0000 7f37aa7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:57:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.962+0000 7f37aa7fc700 10 monclient: dump: 2023-09-30T01:57:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:35.978 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:35.979 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.962+0000 7f37aa7fc700 10 monclient: _finish_auth 0 2023-09-30T01:57:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.962+0000 7f37aa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.962+0000 7f37aa7fc700 1 -- 172.21.15.7:0/1545876537 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ac012ae0 con 0x7f37b40a29e0 2023-09-30T01:57:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.962+0000 7f37bc738700 5 monclient: authenticate success, global_id 4553 2023-09-30T01:57:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.962+0000 7f37bc738700 10 monclient: _renew_subs 2023-09-30T01:57:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.962+0000 7f37bc738700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.962+0000 7f37bc738700 1 -- 172.21.15.7:0/1545876537 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ac009ae0 con 0x7f37b40a29e0 2023-09-30T01:57:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.963+0000 7f37aa7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:35.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.963+0000 7f37aa7fc700 1 -- 172.21.15.7:0/1545876537 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac012e60 con 0x7f37b40a29e0 2023-09-30T01:57:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.963+0000 7f37aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.963+0000 7f37aa7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:57:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.963+0000 7f37aa7fc700 10 monclient: dump: 2023-09-30T01:57:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:35.980 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:35.981 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.963+0000 7f37b8cd1700 4 set_mon_vals no callback set 2023-09-30T01:57:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.963+0000 7f37bc738700 10 monclient: _renew_subs 2023-09-30T01:57:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.963+0000 7f37bc738700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.963+0000 7f37bc738700 1 -- 172.21.15.7:0/1545876537 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37b4142380 con 0x7f37b40a29e0 2023-09-30T01:57:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.963+0000 7f37aa7fc700 1 -- 172.21.15.7:0/1545876537 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f37ac01d630 con 0x7f37b40a29e0 2023-09-30T01:57:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.963+0000 7f37bc738700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.963+0000 7f37bc738700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.963+0000 7f37bc738700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.963+0000 7f37bc738700 1 -- 172.21.15.7:0/1545876537 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37b411cb10 con 0x7f37b40a29e0 2023-09-30T01:57:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.964+0000 7f37aa7fc700 1 --2- 172.21.15.7:0/1545876537 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f37980357e0 0x7f3798037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:35.981 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.964+0000 7f37aa7fc700 1 -- 172.21.15.7:0/1545876537 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f37ac049280 con 0x7f37b40a29e0 2023-09-30T01:57:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.964+0000 7f37ba4d4700 1 --2- 172.21.15.7:0/1545876537 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f37980357e0 0x7f3798037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.964+0000 7f37ba4d4700 10 monclient: get_auth_request con 0x7f37980357e0 auth_method 0 2023-09-30T01:57:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.966+0000 7f37ba4d4700 1 --2- 172.21.15.7:0/1545876537 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f37980357e0 0x7f3798037c90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f37b0007fd0 tx=0x7f37b00020e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.968+0000 7f37aa7fc700 1 -- 172.21.15.7:0/1545876537 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f37b411cb10 con 0x7f37b40a29e0 2023-09-30T01:57:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.968+0000 7f37aa7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:35.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:35.968+0000 7f37aa7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:57:36.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:36.202+0000 7f37bc738700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:57:36.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:36.202+0000 7f37bc738700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:36.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:36.202+0000 7f37bc738700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:36.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:36.202+0000 7f37bc738700 1 -- 172.21.15.7:0/1545876537 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f37b411cb10 con 0x7f37b40a29e0 2023-09-30T01:57:36.202 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:36.202+0000 7f37aa7fc700 1 -- 172.21.15.7:0/1545876537 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f37b411cb10 con 0x7f37b40a29e0 2023-09-30T01:57:36.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:36.202+0000 7f37aa7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:36.203 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:36.202+0000 7f37aa7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:57:36.203 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:57:36.203 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:57:36.204 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:36.204+0000 7f37bc738700 1 -- 172.21.15.7:0/1545876537 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f37980357e0 msgr2=0x7f3798037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:36.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:36.204+0000 7f37bc738700 1 --2- 172.21.15.7:0/1545876537 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f37980357e0 0x7f3798037c90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f37b0007fd0 tx=0x7f37b00020e0 comp rx=0 tx=0).stop 2023-09-30T01:57:36.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:36.204+0000 7f37bc738700 10 monclient: shutdown 2023-09-30T01:57:36.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:36.204+0000 7f37bc738700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:36.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:36.204+0000 7f37bc738700 1 -- 172.21.15.7:0/1545876537 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37b40a29e0 msgr2=0x7f37b409df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:36.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:36.204+0000 7f37bc738700 1 --2- 172.21.15.7:0/1545876537 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37b40a29e0 0x7f37b409df20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f37ac014000 tx=0x7f37ac017000 comp rx=0 tx=0).stop 2023-09-30T01:57:36.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:36.204+0000 7f37bc738700 1 -- 172.21.15.7:0/1545876537 shutdown_connections 2023-09-30T01:57:36.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:36.204+0000 7f37bc738700 1 --2- 172.21.15.7:0/1545876537 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f37980357e0 0x7f3798037c90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:36.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:36.204+0000 7f37bc738700 1 --2- 172.21.15.7:0/1545876537 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37b4097160 0x7f37b41415d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:36.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:36.204+0000 7f37bc738700 1 --2- 172.21.15.7:0/1545876537 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f37b40a29e0 0x7f37b409df20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:36.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:36.204+0000 7f37bc738700 1 --2- 172.21.15.7:0/1545876537 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37b4097530 0x7f37b412e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:36.205 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:36.204+0000 7f37bc738700 1 -- 172.21.15.7:0/1545876537 >> 172.21.15.7:0/1545876537 conn(0x7f37b40710d0 msgr2=0x7f37b4130510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:36.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:36.204+0000 7f37bc738700 1 -- 172.21.15.7:0/1545876537 shutdown_connections 2023-09-30T01:57:36.206 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:36.204+0000 7f37bc738700 1 -- 172.21.15.7:0/1545876537 wait complete. 2023-09-30T01:57:39.216 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:57:39.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.347+0000 7f547b5ea700 10 monclient: get_monmap_and_config 2023-09-30T01:57:39.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.347+0000 7f547b5ea700 10 monclient: build_initial_monmap 2023-09-30T01:57:39.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.347+0000 7f547b5ea700 10 monclient: monmap: 2023-09-30T01:57:39.348 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:39.348 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:39.348 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:57:39.349004+0000 2023-09-30T01:57:39.348 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:57:39.349004+0000 2023-09-30T01:57:39.348 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:39.348 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:39.348 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:57:39.349 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:57:39.349 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:57:39.349 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:57:39.349 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:57:39.349 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:39.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.348+0000 7f547b5ea700 1 Processor -- start 2023-09-30T01:57:39.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.348+0000 7f547b5ea700 1 -- start start 2023-09-30T01:57:39.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.348+0000 7f547b5ea700 10 monclient: init 2023-09-30T01:57:39.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f547b5ea700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:39.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f547b5ea700 10 monclient: _add_conns ranks=[1,4,3,0,2] 2023-09-30T01:57:39.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f547b5ea700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5474096a50 addr v1:172.21.15.50:6789/0 2023-09-30T01:57:39.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f547b5ea700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f54740a2390 0x7f5474096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f547b5ea700 10 monclient(hunting): picked mon.noname-c con 0x7f54740a2390 addr v2:172.21.15.50:3300/0 2023-09-30T01:57:39.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f547b5ea700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5474097360 0x7f5474139720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f547b5ea700 10 monclient(hunting): picked mon.noname-a con 0x7f5474097360 addr v2:172.21.15.7:3300/0 2023-09-30T01:57:39.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f547b5ea700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54740980e0 con 0x7f5474096a50 2023-09-30T01:57:39.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f547b5ea700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:39.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f547b5ea700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5474132eb0 con 0x7f5474097360 2023-09-30T01:57:39.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f547b5ea700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:39.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f547b5ea700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54740991a0 con 0x7f54740a2390 2023-09-30T01:57:39.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f547b5ea700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:39.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f547b5ea700 10 monclient(hunting): authenticate will time out at 1348.414277s 2023-09-30T01:57:39.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5479b87700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5474097360 0x7f5474139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5479b87700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5474097360 0x7f5474139720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49974/0 (socket says 172.21.15.7:49974) 2023-09-30T01:57:39.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5479b87700 1 -- 172.21.15.7:0/2282239078 learned_addr learned my addr 172.21.15.7:0/2282239078 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:39.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5479b87700 10 monclient(hunting): get_auth_request con 0x7f5474097360 auth_method 0 2023-09-30T01:57:39.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5479b87700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:39.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5479b87700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:39.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5479b87700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:39.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5479b87700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:39.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5479b87700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:39.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5479b87700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:39.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5478b85700 1 --2- 172.21.15.7:0/2282239078 >> v2:172.21.15.50:3300/0 conn(0x7f54740a2390 0x7f5474096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5478b85700 10 monclient(hunting): get_auth_request con 0x7f54740a2390 auth_method 0 2023-09-30T01:57:39.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5478b85700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:39.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5478b85700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:39.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5478b85700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:39.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5479b87700 10 monclient(hunting): handle_auth_done global_id 4536 payload 274 2023-09-30T01:57:39.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5479b87700 1 -- 172.21.15.7:0/2282239078 >> v2:172.21.15.50:3300/0 conn(0x7f54740a2390 msgr2=0x7f5474096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:39.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5479b87700 1 --2- 172.21.15.7:0/2282239078 >> v2:172.21.15.50:3300/0 conn(0x7f54740a2390 0x7f5474096e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5479b87700 1 -- 172.21.15.7:0/2282239078 >> v1:172.21.15.50:6789/0 conn(0x7f5474096a50 legacy=0x7f54741327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:39.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5479b87700 10 monclient: _finish_hunting 0 2023-09-30T01:57:39.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5479b87700 1 monclient: found mon.noname-a 2023-09-30T01:57:39.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5479b87700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:39.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5479b87700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:57:39.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.349+0000 7f5479b87700 1 -- 172.21.15.7:0/2282239078 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5474099480 con 0x7f5474097360 2023-09-30T01:57:39.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.351+0000 7f5479b87700 1 -- 172.21.15.7:0/2282239078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5474097360 msgr2=0x7f5474139720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:39.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.352+0000 7f5479b87700 1 --2- 172.21.15.7:0/2282239078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5474097360 0x7f5474139720 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f547406a070 tx=0x7f546c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3345eeef9b7c579f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:39.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f546bfff700 1 -- 172.21.15.7:0/2282239078 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c00e070 con 0x7f5474097360 2023-09-30T01:57:39.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f546bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:39.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f546bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:57:39.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f546bfff700 10 monclient: dump: 2023-09-30T01:57:39.362 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:39.362 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:39.363 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:39.363 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:39.363 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:39.363 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:39.363 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:39.363 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:39.363 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:39.363 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:39.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f546bfff700 10 monclient: _finish_auth 0 2023-09-30T01:57:39.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f546bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:39.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f547b5ea700 5 monclient: authenticate success, global_id 4536 2023-09-30T01:57:39.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f547b5ea700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:57:39.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f546bfff700 1 -- 172.21.15.7:0/2282239078 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546c00bc20 con 0x7f5474097360 2023-09-30T01:57:39.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f546bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:39.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f547b5ea700 10 monclient: get_monmap_and_config success 2023-09-30T01:57:39.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f547b5ea700 4 set_mon_vals no callback set 2023-09-30T01:57:39.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f546bfff700 1 -- 172.21.15.7:0/2282239078 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c0092e0 con 0x7f5474097360 2023-09-30T01:57:39.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f546bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:39.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f546bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:57:39.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f546bfff700 10 monclient: dump: 2023-09-30T01:57:39.366 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:39.366 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:39.366 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:39.366 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:39.366 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:39.366 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:39.366 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:39.367 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:39.367 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:39.367 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:39.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f547b5ea700 10 monclient: shutdown 2023-09-30T01:57:39.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f547b5ea700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:39.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f547b5ea700 1 -- 172.21.15.7:0/2282239078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5474097360 msgr2=0x7f5474139720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f547b5ea700 1 --2- 172.21.15.7:0/2282239078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5474097360 0x7f5474139720 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f547406a070 tx=0x7f546c009840 comp rx=0 tx=0).stop 2023-09-30T01:57:39.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f547b5ea700 1 -- 172.21.15.7:0/2282239078 shutdown_connections 2023-09-30T01:57:39.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f547b5ea700 1 --2- 172.21.15.7:0/2282239078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5474097360 0x7f5474139720 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f547b5ea700 1 --2- 172.21.15.7:0/2282239078 >> v2:172.21.15.50:3300/0 conn(0x7f54740a2390 0x7f5474096e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.353+0000 7f547b5ea700 1 -- 172.21.15.7:0/2282239078 >> 172.21.15.7:0/2282239078 conn(0x7f5474070cf0 msgr2=0x7f54740710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:39.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.354+0000 7f547b5ea700 1 -- 172.21.15.7:0/2282239078 shutdown_connections 2023-09-30T01:57:39.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.354+0000 7f547b5ea700 1 -- 172.21.15.7:0/2282239078 wait complete. 2023-09-30T01:57:39.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.354+0000 7f547b5ea700 10 monclient: build_initial_monmap 2023-09-30T01:57:39.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.354+0000 7f547b5ea700 10 monclient: monmap: 2023-09-30T01:57:39.368 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:39.368 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:57:39.368 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:57:39.368 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:57:39.368 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:39.368 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:39.369 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:57:39.369 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:57:39.369 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:57:39.369 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:39.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.354+0000 7f547b5ea700 1 Processor -- start 2023-09-30T01:57:39.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.354+0000 7f547b5ea700 1 -- start start 2023-09-30T01:57:39.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.354+0000 7f547b5ea700 10 monclient: init 2023-09-30T01:57:39.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f547b5ea700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:39.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f547b5ea700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:57:39.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f547b5ea700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5474096a50 0x7f547409b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f547b5ea700 10 monclient(hunting): picked mon.noname-b con 0x7f5474096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:57:39.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f547b5ea700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5474097360 0x7f547409bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f547b5ea700 10 monclient(hunting): picked mon.noname-c con 0x7f5474097360 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:39.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f547b5ea700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54740a2390 0x7f547409c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f547b5ea700 10 monclient(hunting): picked mon.noname-a con 0x7f54740a2390 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:57:39.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f547b5ea700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:39.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f547b5ea700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547409deb0 con 0x7f54740a2390 2023-09-30T01:57:39.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f547b5ea700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:39.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f547b5ea700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547409eda0 con 0x7f5474096a50 2023-09-30T01:57:39.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f547b5ea700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:39.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f547b5ea700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f547409d860 con 0x7f5474097360 2023-09-30T01:57:39.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f547b5ea700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:39.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f547b5ea700 10 monclient(hunting): authenticate will time out at 1348.420607s 2023-09-30T01:57:39.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5479b87700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54740a2390 0x7f547409c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5478b85700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5474097360 0x7f547409bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5479b87700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54740a2390 0x7f547409c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49990/0 (socket says 172.21.15.7:49990) 2023-09-30T01:57:39.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5479b87700 1 -- 172.21.15.7:0/2892821373 learned_addr learned my addr 172.21.15.7:0/2892821373 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:39.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5479b87700 10 monclient(hunting): get_auth_request con 0x7f54740a2390 auth_method 0 2023-09-30T01:57:39.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5479b87700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:39.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5479b87700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:39.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5479b87700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:39.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5478b85700 10 monclient(hunting): get_auth_request con 0x7f5474097360 auth_method 0 2023-09-30T01:57:39.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5478b85700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:39.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5478b85700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:39.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5478b85700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:39.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5479386700 1 --2- 172.21.15.7:0/2892821373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5474096a50 0x7f547409b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5479386700 10 monclient(hunting): get_auth_request con 0x7f5474096a50 auth_method 0 2023-09-30T01:57:39.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5479386700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:39.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5479386700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:39.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5479386700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5479b87700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5479b87700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5479b87700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5478b85700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5478b85700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5478b85700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5479386700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5479386700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.355+0000 7f5479386700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:39.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.356+0000 7f5479b87700 10 monclient(hunting): handle_auth_done global_id 4539 payload 824 2023-09-30T01:57:39.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.356+0000 7f5479b87700 1 -- 172.21.15.7:0/2892821373 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5474097360 msgr2=0x7f547409bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.356+0000 7f5479b87700 1 --2- 172.21.15.7:0/2892821373 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5474097360 0x7f547409bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.356+0000 7f5479b87700 1 -- 172.21.15.7:0/2892821373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5474096a50 msgr2=0x7f547409b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.356+0000 7f5479b87700 1 --2- 172.21.15.7:0/2892821373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5474096a50 0x7f547409b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.356+0000 7f5479b87700 10 monclient: _finish_hunting 0 2023-09-30T01:57:39.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.356+0000 7f5479b87700 1 monclient: found mon.noname-a 2023-09-30T01:57:39.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.356+0000 7f5479b87700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:39.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.356+0000 7f5479b87700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:57:39.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.356+0000 7f5479b87700 1 -- 172.21.15.7:0/2892821373 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546c009ae0 con 0x7f54740a2390 2023-09-30T01:57:39.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.356+0000 7f5478b85700 1 --2- 172.21.15.7:0/2892821373 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5474097360 0x7f547409bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:39.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f5479b87700 1 -- 172.21.15.7:0/2892821373 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54740a2390 msgr2=0x7f547409c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:39.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f5479b87700 1 --2- 172.21.15.7:0/2892821373 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54740a2390 0x7f547409c150 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f546c009fd0 tx=0x7f546c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:39.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f54697fa700 1 -- 172.21.15.7:0/2892821373 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c0071b0 con 0x7f54740a2390 2023-09-30T01:57:39.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f54697fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:39.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f54697fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:57:39.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f54697fa700 10 monclient: dump: 2023-09-30T01:57:39.376 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:39.376 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:39.376 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:39.376 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:39.376 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:39.376 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:39.376 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:39.376 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:39.376 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:39.376 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:39.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f54697fa700 10 monclient: _finish_auth 0 2023-09-30T01:57:39.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f54697fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:39.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f547b5ea700 5 monclient: authenticate success, global_id 4539 2023-09-30T01:57:39.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f54697fa700 1 -- 172.21.15.7:0/2892821373 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546c012be0 con 0x7f54740a2390 2023-09-30T01:57:39.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f54697fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:39.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f54697fa700 1 -- 172.21.15.7:0/2892821373 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c01d420 con 0x7f54740a2390 2023-09-30T01:57:39.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f547b5ea700 10 monclient: _renew_subs 2023-09-30T01:57:39.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f547b5ea700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:39.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f547b5ea700 1 -- 172.21.15.7:0/2892821373 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54741304f0 con 0x7f54740a2390 2023-09-30T01:57:39.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f54697fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:39.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f546b7fe700 4 set_mon_vals no callback set 2023-09-30T01:57:39.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f54697fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:57:39.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f54697fa700 10 monclient: dump: 2023-09-30T01:57:39.378 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:39.378 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:39.378 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:39.378 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:39.378 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:39.378 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:39.378 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:39.378 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:39.378 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:39.378 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:39.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f547b5ea700 10 monclient: _renew_subs 2023-09-30T01:57:39.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f547b5ea700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:39.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.358+0000 7f547b5ea700 1 -- 172.21.15.7:0/2892821373 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54740a1b50 con 0x7f54740a2390 2023-09-30T01:57:39.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.359+0000 7f54697fa700 1 -- 172.21.15.7:0/2892821373 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f546c01d7d0 con 0x7f54740a2390 2023-09-30T01:57:39.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.359+0000 7f547b5ea700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:39.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.359+0000 7f547b5ea700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:39.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.359+0000 7f547b5ea700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:39.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.359+0000 7f547b5ea700 1 -- 172.21.15.7:0/2892821373 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54740a4ba0 con 0x7f54740a2390 2023-09-30T01:57:39.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.359+0000 7f54697fa700 1 --2- 172.21.15.7:0/2892821373 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5458035810 0x7f5458037cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:39.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.359+0000 7f54697fa700 1 -- 172.21.15.7:0/2892821373 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f546c049320 con 0x7f54740a2390 2023-09-30T01:57:39.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.360+0000 7f5479386700 1 --2- 172.21.15.7:0/2892821373 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5458035810 0x7f5458037cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:39.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.360+0000 7f5479386700 10 monclient: get_auth_request con 0x7f5458035810 auth_method 0 2023-09-30T01:57:39.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.362+0000 7f5479386700 1 --2- 172.21.15.7:0/2892821373 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5458035810 0x7f5458037cc0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5470003a30 tx=0x7f5470007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:39.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.364+0000 7f54697fa700 1 -- 172.21.15.7:0/2892821373 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f546c0076e0 con 0x7f54740a2390 2023-09-30T01:57:39.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.364+0000 7f54697fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:39.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.364+0000 7f54697fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:57:39.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.601+0000 7f547b5ea700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:57:39.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.601+0000 7f547b5ea700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:39.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.601+0000 7f547b5ea700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:39.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.601+0000 7f547b5ea700 1 -- 172.21.15.7:0/2892821373 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5474064c10 con 0x7f54740a2390 2023-09-30T01:57:39.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.601+0000 7f54697fa700 1 -- 172.21.15.7:0/2892821373 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f546c02a080 con 0x7f54740a2390 2023-09-30T01:57:39.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.601+0000 7f54697fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:39.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.601+0000 7f54697fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:57:39.602 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:57:39.602 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:57:39.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.603+0000 7f547b5ea700 1 -- 172.21.15.7:0/2892821373 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5458035810 msgr2=0x7f5458037cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.603+0000 7f547b5ea700 1 --2- 172.21.15.7:0/2892821373 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5458035810 0x7f5458037cc0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5470003a30 tx=0x7f5470007ea0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.603+0000 7f547b5ea700 10 monclient: shutdown 2023-09-30T01:57:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.603+0000 7f547b5ea700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.603+0000 7f547b5ea700 1 -- 172.21.15.7:0/2892821373 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54740a2390 msgr2=0x7f547409c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.603+0000 7f547b5ea700 1 --2- 172.21.15.7:0/2892821373 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54740a2390 0x7f547409c150 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f546c009fd0 tx=0x7f546c017000 comp rx=0 tx=0).stop 2023-09-30T01:57:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.603+0000 7f547b5ea700 1 -- 172.21.15.7:0/2892821373 shutdown_connections 2023-09-30T01:57:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.603+0000 7f547b5ea700 1 --2- 172.21.15.7:0/2892821373 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5458035810 0x7f5458037cc0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.603+0000 7f547b5ea700 1 --2- 172.21.15.7:0/2892821373 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54740a2390 0x7f547409c150 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.603+0000 7f547b5ea700 1 --2- 172.21.15.7:0/2892821373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5474096a50 0x7f547409b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.603+0000 7f547b5ea700 1 --2- 172.21.15.7:0/2892821373 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f5474097360 0x7f547409bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.603+0000 7f547b5ea700 1 -- 172.21.15.7:0/2892821373 >> 172.21.15.7:0/2892821373 conn(0x7f5474070cf0 msgr2=0x7f54741314a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:39.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.603+0000 7f547b5ea700 1 -- 172.21.15.7:0/2892821373 shutdown_connections 2023-09-30T01:57:39.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:39.603+0000 7f547b5ea700 1 -- 172.21.15.7:0/2892821373 wait complete. 2023-09-30T01:57:42.615 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:57:42.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.746+0000 7f54f863e700 10 monclient: get_monmap_and_config 2023-09-30T01:57:42.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.746+0000 7f54f863e700 10 monclient: build_initial_monmap 2023-09-30T01:57:42.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.746+0000 7f54f863e700 10 monclient: monmap: 2023-09-30T01:57:42.747 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:42.747 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:42.747 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:57:42.747571+0000 2023-09-30T01:57:42.747 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:57:42.747571+0000 2023-09-30T01:57:42.747 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:42.747 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:42.747 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:57:42.747 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:57:42.747 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:57:42.747 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:57:42.747 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:57:42.747 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:42.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.747+0000 7f54f863e700 1 Processor -- start 2023-09-30T01:57:42.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.747+0000 7f54f863e700 1 -- start start 2023-09-30T01:57:42.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.747+0000 7f54f863e700 10 monclient: init 2023-09-30T01:57:42.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.747+0000 7f54f863e700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:42.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.747+0000 7f54f863e700 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-09-30T01:57:42.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.747+0000 7f54f863e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54f00a29e0 addr v1:172.21.15.7:6789/0 2023-09-30T01:57:42.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.747+0000 7f54f863e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f54f0097160 0x7f54f0137550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:42.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.747+0000 7f54f863e700 10 monclient(hunting): picked mon.noname-a con 0x7f54f0097160 addr v2:172.21.15.7:3300/0 2023-09-30T01:57:42.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.747+0000 7f54f863e700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f54f0097560 0x7f54f0139aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:42.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.747+0000 7f54f863e700 10 monclient(hunting): picked mon.noname-b con 0x7f54f0097560 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:42.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.747+0000 7f54f863e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f0069cc0 con 0x7f54f00a29e0 2023-09-30T01:57:42.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.747+0000 7f54f863e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:42.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.747+0000 7f54f863e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54f0098ea0 con 0x7f54f0097160 2023-09-30T01:57:42.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.747+0000 7f54f863e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:42.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.747+0000 7f54f863e700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54f00a2db0 con 0x7f54f0097560 2023-09-30T01:57:42.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.747+0000 7f54f863e700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:42.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.747+0000 7f54f863e700 10 monclient(hunting): authenticate will time out at 1351.812959s 2023-09-30T01:57:42.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f63da700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f54f00a29e0 0x7f54f0096a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38410/0 (socket says 172.21.15.7:38410) 2023-09-30T01:57:42.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f63da700 1 -- 172.21.15.7:0/338866352 learned_addr learned my addr 172.21.15.7:0/338866352 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:57:42.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f6bdb700 1 --2- 172.21.15.7:0/338866352 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f54f0097560 0x7f54f0139aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:42.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f5bd9700 1 --2- 172.21.15.7:0/338866352 >> v2:172.21.15.7:3300/0 conn(0x7f54f0097160 0x7f54f0137550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:42.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f6bdb700 10 monclient(hunting): get_auth_request con 0x7f54f0097560 auth_method 0 2023-09-30T01:57:42.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f6bdb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:42.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f6bdb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:42.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f6bdb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:42.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f5bd9700 10 monclient(hunting): get_auth_request con 0x7f54f0097160 auth_method 0 2023-09-30T01:57:42.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f5bd9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:42.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f5bd9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:42.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f5bd9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:42.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f6bdb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:42.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f6bdb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:42.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f6bdb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:42.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f5bd9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:42.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f5bd9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:42.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f5bd9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:42.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f6bdb700 10 monclient(hunting): handle_auth_done global_id 4559 payload 274 2023-09-30T01:57:42.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f6bdb700 1 -- 172.21.15.7:0/338866352 >> v2:172.21.15.7:3300/0 conn(0x7f54f0097160 msgr2=0x7f54f0137550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:42.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f6bdb700 1 --2- 172.21.15.7:0/338866352 >> v2:172.21.15.7:3300/0 conn(0x7f54f0097160 0x7f54f0137550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:42.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f6bdb700 1 -- 172.21.15.7:0/338866352 >> v1:172.21.15.7:6789/0 conn(0x7f54f00a29e0 legacy=0x7f54f0096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:42.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f6bdb700 10 monclient: _finish_hunting 0 2023-09-30T01:57:42.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f6bdb700 1 monclient: found mon.noname-b 2023-09-30T01:57:42.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f6bdb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:42.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f6bdb700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:57:42.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f6bdb700 1 -- 172.21.15.7:0/338866352 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f0098270 con 0x7f54f0097560 2023-09-30T01:57:42.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.748+0000 7f54f5bd9700 1 --2- 172.21.15.7:0/338866352 >> v2:172.21.15.7:3300/0 conn(0x7f54f0097160 0x7f54f0137550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:42.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.750+0000 7f54f6bdb700 1 -- 172.21.15.7:0/338866352 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f54f0097560 msgr2=0x7f54f0139aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:42.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.750+0000 7f54f6bdb700 1 --2- 172.21.15.7:0/338866352 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f54f0097560 0x7f54f0139aa0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f54e8009e60 tx=0x7f54e8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=571a7b2bbab07eaa server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:42.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.750+0000 7f54f53d8700 1 -- 172.21.15.7:0/338866352 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e800e070 con 0x7f54f0097560 2023-09-30T01:57:42.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.750+0000 7f54f53d8700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:42.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.750+0000 7f54f53d8700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:57:42.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.750+0000 7f54f53d8700 10 monclient: dump: 2023-09-30T01:57:42.756 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:42.756 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:42.756 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:42.756 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:42.756 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:42.756 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:42.756 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:42.756 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:42.756 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:42.757 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:42.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.750+0000 7f54f53d8700 10 monclient: _finish_auth 0 2023-09-30T01:57:42.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.750+0000 7f54f53d8700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:42.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.750+0000 7f54f863e700 5 monclient: authenticate success, global_id 4559 2023-09-30T01:57:42.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.750+0000 7f54f863e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:57:42.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.750+0000 7f54f53d8700 1 -- 172.21.15.7:0/338866352 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e800bd30 con 0x7f54f0097560 2023-09-30T01:57:42.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.750+0000 7f54f53d8700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:42.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.750+0000 7f54f863e700 10 monclient: get_monmap_and_config success 2023-09-30T01:57:42.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.750+0000 7f54f863e700 4 set_mon_vals no callback set 2023-09-30T01:57:42.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.750+0000 7f54f53d8700 1 -- 172.21.15.7:0/338866352 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e800bea0 con 0x7f54f0097560 2023-09-30T01:57:42.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.750+0000 7f54f53d8700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:42.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.750+0000 7f54f53d8700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:57:42.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.750+0000 7f54f53d8700 10 monclient: dump: 2023-09-30T01:57:42.758 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:42.758 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:42.760 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:42.760 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:42.761 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:42.761 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:42.761 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:42.761 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:42.761 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:42.761 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:42.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.751+0000 7f54f863e700 10 monclient: shutdown 2023-09-30T01:57:42.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.751+0000 7f54f863e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:42.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.751+0000 7f54f863e700 1 -- 172.21.15.7:0/338866352 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f54f0097560 msgr2=0x7f54f0139aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:42.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.751+0000 7f54f863e700 1 --2- 172.21.15.7:0/338866352 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f54f0097560 0x7f54f0139aa0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f54e8009e60 tx=0x7f54e8009780 comp rx=0 tx=0).stop 2023-09-30T01:57:42.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.751+0000 7f54f863e700 1 -- 172.21.15.7:0/338866352 shutdown_connections 2023-09-30T01:57:42.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.751+0000 7f54f863e700 1 --2- 172.21.15.7:0/338866352 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f54f0097560 0x7f54f0139aa0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:42.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.751+0000 7f54f863e700 1 --2- 172.21.15.7:0/338866352 >> v2:172.21.15.7:3300/0 conn(0x7f54f0097160 0x7f54f0137550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:42.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.751+0000 7f54f863e700 1 -- 172.21.15.7:0/338866352 >> 172.21.15.7:0/338866352 conn(0x7f54f00710d0 msgr2=0x7f54f00714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:42.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.751+0000 7f54f863e700 1 -- 172.21.15.7:0/338866352 shutdown_connections 2023-09-30T01:57:42.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.751+0000 7f54f863e700 1 -- 172.21.15.7:0/338866352 wait complete. 2023-09-30T01:57:42.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.752+0000 7f54f863e700 10 monclient: build_initial_monmap 2023-09-30T01:57:42.766 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.752+0000 7f54f863e700 10 monclient: monmap: 2023-09-30T01:57:42.766 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:57:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:57:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:57:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:57:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:57:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:57:42.767 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.752+0000 7f54f863e700 1 Processor -- start 2023-09-30T01:57:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.752+0000 7f54f863e700 1 -- start start 2023-09-30T01:57:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.752+0000 7f54f863e700 10 monclient: init 2023-09-30T01:57:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.752+0000 7f54f863e700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.752+0000 7f54f863e700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:57:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.752+0000 7f54f863e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54f0097160 0x7f54f012df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.752+0000 7f54f863e700 10 monclient(hunting): picked mon.noname-b con 0x7f54f0097160 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:57:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f863e700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f54f0097560 0x7f54f009cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f863e700 10 monclient(hunting): picked mon.noname-c con 0x7f54f0097560 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f863e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54f00a29e0 0x7f54f009d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f863e700 10 monclient(hunting): picked mon.noname-a con 0x7f54f00a29e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:57:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f863e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f863e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f00a1220 con 0x7f54f00a29e0 2023-09-30T01:57:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f863e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f863e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f0142560 con 0x7f54f0097160 2023-09-30T01:57:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f863e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:42.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f863e700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54f0135130 con 0x7f54f0097560 2023-09-30T01:57:42.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f863e700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:42.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f863e700 10 monclient(hunting): authenticate will time out at 1351.818064s 2023-09-30T01:57:42.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f5bd9700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f54f0097560 0x7f54f009cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:42.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f5bd9700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f54f0097560 0x7f54f009cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:35888/0 (socket says 172.21.15.7:35888) 2023-09-30T01:57:42.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f6bdb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54f00a29e0 0x7f54f009d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:42.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f5bd9700 1 -- 172.21.15.7:0/705255591 learned_addr learned my addr 172.21.15.7:0/705255591 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:42.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f5bd9700 10 monclient(hunting): get_auth_request con 0x7f54f0097560 auth_method 0 2023-09-30T01:57:42.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f5bd9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:42.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f5bd9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:42.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f5bd9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:42.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f6bdb700 10 monclient(hunting): get_auth_request con 0x7f54f00a29e0 auth_method 0 2023-09-30T01:57:42.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f6bdb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:42.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f6bdb700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:42.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f6bdb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:42.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f5bd9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:42.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f5bd9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:42.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f5bd9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:42.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f6bdb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:42.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f6bdb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:42.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f6bdb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:42.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f63da700 1 --2- 172.21.15.7:0/705255591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54f0097160 0x7f54f012df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:42.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f5bd9700 10 monclient(hunting): handle_auth_done global_id 4562 payload 824 2023-09-30T01:57:42.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f5bd9700 1 -- 172.21.15.7:0/705255591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54f0097160 msgr2=0x7f54f012df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:42.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f5bd9700 1 --2- 172.21.15.7:0/705255591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54f0097160 0x7f54f012df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:42.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f5bd9700 1 -- 172.21.15.7:0/705255591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54f00a29e0 msgr2=0x7f54f009d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:42.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f5bd9700 1 --2- 172.21.15.7:0/705255591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54f00a29e0 0x7f54f009d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:42.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f5bd9700 10 monclient: _finish_hunting 0 2023-09-30T01:57:42.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f5bd9700 1 monclient: found mon.noname-c 2023-09-30T01:57:42.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f5bd9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:42.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f5bd9700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:57:42.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f5bd9700 1 -- 172.21.15.7:0/705255591 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ec009420 con 0x7f54f0097560 2023-09-30T01:57:42.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f63da700 10 monclient: get_auth_request con 0x7f54f0097160 auth_method 0 2023-09-30T01:57:42.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f63da700 1 --2- 172.21.15.7:0/705255591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54f0097160 0x7f54f012df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:57:42.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.753+0000 7f54f6bdb700 1 --2- 172.21.15.7:0/705255591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54f00a29e0 0x7f54f009d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:42.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54f5bd9700 1 -- 172.21.15.7:0/705255591 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f54f0097560 msgr2=0x7f54f009cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:42.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54f5bd9700 1 --2- 172.21.15.7:0/705255591 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f54f0097560 0x7f54f009cc80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f54ec00ec80 tx=0x7f54ec0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:42.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54e27fc700 1 -- 172.21.15.7:0/705255591 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ec00ca00 con 0x7f54f0097560 2023-09-30T01:57:42.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:42.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54e27fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:57:42.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54e27fc700 10 monclient: dump: 2023-09-30T01:57:42.772 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:42.772 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:42.772 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:42.772 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:42.772 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:42.772 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:42.772 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:42.772 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:42.773 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:42.773 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:42.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54e27fc700 10 monclient: _finish_auth 0 2023-09-30T01:57:42.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54e27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:42.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54e27fc700 1 -- 172.21.15.7:0/705255591 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54ec00cb60 con 0x7f54f0097560 2023-09-30T01:57:42.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54f863e700 5 monclient: authenticate success, global_id 4562 2023-09-30T01:57:42.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54f863e700 10 monclient: _renew_subs 2023-09-30T01:57:42.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54f863e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:42.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54f863e700 1 -- 172.21.15.7:0/705255591 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e8009a20 con 0x7f54f0097560 2023-09-30T01:57:42.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54e27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:42.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54e27fc700 1 -- 172.21.15.7:0/705255591 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ec00cee0 con 0x7f54f0097560 2023-09-30T01:57:42.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54e27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:42.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54e27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:57:42.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54e27fc700 10 monclient: dump: 2023-09-30T01:57:42.774 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:42.774 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:42.774 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:42.774 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:42.774 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:42.774 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:42.774 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:42.774 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:42.774 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:42.774 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:42.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54f4bd7700 4 set_mon_vals no callback set 2023-09-30T01:57:42.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54f863e700 10 monclient: _renew_subs 2023-09-30T01:57:42.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54f863e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:42.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.755+0000 7f54f863e700 1 -- 172.21.15.7:0/705255591 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f009e820 con 0x7f54f0097560 2023-09-30T01:57:42.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.756+0000 7f54e27fc700 1 -- 172.21.15.7:0/705255591 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f54ec013680 con 0x7f54f0097560 2023-09-30T01:57:42.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.756+0000 7f54f863e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:42.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.756+0000 7f54f863e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:42.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.756+0000 7f54f863e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:42.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.756+0000 7f54f863e700 1 -- 172.21.15.7:0/705255591 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f00a0870 con 0x7f54f0097560 2023-09-30T01:57:42.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.756+0000 7f54e27fc700 1 --2- 172.21.15.7:0/705255591 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f54d40357e0 0x7f54d4037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:42.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.757+0000 7f54e27fc700 1 -- 172.21.15.7:0/705255591 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f54ec048000 con 0x7f54f0097560 2023-09-30T01:57:42.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.757+0000 7f54f63da700 1 --2- 172.21.15.7:0/705255591 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f54d40357e0 0x7f54d4037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:42.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.757+0000 7f54f63da700 10 monclient: get_auth_request con 0x7f54d40357e0 auth_method 0 2023-09-30T01:57:42.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.759+0000 7f54f63da700 1 --2- 172.21.15.7:0/705255591 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f54d40357e0 0x7f54d4037c90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f54e4002870 tx=0x7f54e4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:42.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.761+0000 7f54e27fc700 1 -- 172.21.15.7:0/705255591 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f54f00a0870 con 0x7f54f0097560 2023-09-30T01:57:42.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.761+0000 7f54e27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:42.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.761+0000 7f54e27fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:57:42.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.995+0000 7f54f863e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:57:42.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.995+0000 7f54f863e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:42.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.995+0000 7f54f863e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:42.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.995+0000 7f54f863e700 1 -- 172.21.15.7:0/705255591 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f54f00a0a50 con 0x7f54f0097560 2023-09-30T01:57:42.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.996+0000 7f54e27fc700 1 -- 172.21.15.7:0/705255591 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f54f00a0a50 con 0x7f54f0097560 2023-09-30T01:57:42.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.996+0000 7f54e27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:42.996 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.996+0000 7f54e27fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:57:42.997 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:57:42.997 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:57:42.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.998+0000 7f54f863e700 1 -- 172.21.15.7:0/705255591 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f54d40357e0 msgr2=0x7f54d4037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:42.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.998+0000 7f54f863e700 1 --2- 172.21.15.7:0/705255591 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f54d40357e0 0x7f54d4037c90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f54e4002870 tx=0x7f54e4009040 comp rx=0 tx=0).stop 2023-09-30T01:57:42.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.998+0000 7f54f863e700 10 monclient: shutdown 2023-09-30T01:57:42.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.998+0000 7f54f863e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:42.998 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.998+0000 7f54f863e700 1 -- 172.21.15.7:0/705255591 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f54f0097560 msgr2=0x7f54f009cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:42.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.998+0000 7f54f863e700 1 --2- 172.21.15.7:0/705255591 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f54f0097560 0x7f54f009cc80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f54ec00ec80 tx=0x7f54ec0091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:42.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.998+0000 7f54f863e700 1 -- 172.21.15.7:0/705255591 shutdown_connections 2023-09-30T01:57:42.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.998+0000 7f54f863e700 1 --2- 172.21.15.7:0/705255591 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f54d40357e0 0x7f54d4037c90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:42.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.998+0000 7f54f863e700 1 --2- 172.21.15.7:0/705255591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54f00a29e0 0x7f54f009d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:42.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.998+0000 7f54f863e700 1 --2- 172.21.15.7:0/705255591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f54f0097160 0x7f54f012df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:42.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.998+0000 7f54f863e700 1 --2- 172.21.15.7:0/705255591 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f54f0097560 0x7f54f009cc80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:42.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.998+0000 7f54f863e700 1 -- 172.21.15.7:0/705255591 >> 172.21.15.7:0/705255591 conn(0x7f54f00710d0 msgr2=0x7f54f0134030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:42.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.998+0000 7f54f863e700 1 -- 172.21.15.7:0/705255591 shutdown_connections 2023-09-30T01:57:42.999 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:42.998+0000 7f54f863e700 1 -- 172.21.15.7:0/705255591 wait complete. 2023-09-30T01:57:46.011 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:57:46.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.141+0000 7fa837017700 10 monclient: get_monmap_and_config 2023-09-30T01:57:46.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.141+0000 7fa837017700 10 monclient: build_initial_monmap 2023-09-30T01:57:46.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.141+0000 7fa837017700 10 monclient: monmap: 2023-09-30T01:57:46.141 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:46.141 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:46.141 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:57:46.142095+0000 2023-09-30T01:57:46.141 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:57:46.142095+0000 2023-09-30T01:57:46.142 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:46.142 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:46.142 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:57:46.142 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:57:46.142 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:57:46.142 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:57:46.142 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:57:46.142 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:46.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.141+0000 7fa837017700 1 Processor -- start 2023-09-30T01:57:46.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.141+0000 7fa837017700 1 -- start start 2023-09-30T01:57:46.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.141+0000 7fa837017700 10 monclient: init 2023-09-30T01:57:46.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa837017700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:46.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa837017700 10 monclient: _add_conns ranks=[1,3,4,0,2] 2023-09-30T01:57:46.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa837017700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa830096a50 addr v1:172.21.15.50:6789/0 2023-09-30T01:57:46.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa837017700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa8300a2390 0x7fa830096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:46.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa837017700 10 monclient(hunting): picked mon.noname-a con 0x7fa8300a2390 addr v2:172.21.15.7:3300/0 2023-09-30T01:57:46.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa837017700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa830097360 0x7fa830139720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:46.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa837017700 10 monclient(hunting): picked mon.noname-c con 0x7fa830097360 addr v2:172.21.15.50:3300/0 2023-09-30T01:57:46.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa837017700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8300980e0 con 0x7fa830096a50 2023-09-30T01:57:46.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa837017700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:46.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa837017700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa830132eb0 con 0x7fa8300a2390 2023-09-30T01:57:46.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa837017700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:46.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa837017700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8300991a0 con 0x7fa830097360 2023-09-30T01:57:46.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa837017700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:46.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa837017700 10 monclient(hunting): authenticate will time out at 1355.207345s 2023-09-30T01:57:46.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa82ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa8300a2390 0x7fa830096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:46.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa82ffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa8300a2390 0x7fa830096e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52300/0 (socket says 172.21.15.7:52300) 2023-09-30T01:57:46.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa82ffff700 1 -- 172.21.15.7:0/3751390207 learned_addr learned my addr 172.21.15.7:0/3751390207 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:46.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa82ffff700 10 monclient(hunting): get_auth_request con 0x7fa8300a2390 auth_method 0 2023-09-30T01:57:46.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa82ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:46.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa82ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:46.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa82ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:46.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa82ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:46.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa82ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:46.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa82ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:46.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa8355b4700 1 --2- 172.21.15.7:0/3751390207 >> v2:172.21.15.50:3300/0 conn(0x7fa830097360 0x7fa830139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:46.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa8355b4700 10 monclient(hunting): get_auth_request con 0x7fa830097360 auth_method 0 2023-09-30T01:57:46.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa8355b4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:46.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa8355b4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:46.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa8355b4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:46.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa82ffff700 10 monclient(hunting): handle_auth_done global_id 4548 payload 274 2023-09-30T01:57:46.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa82ffff700 1 -- 172.21.15.7:0/3751390207 >> v2:172.21.15.50:3300/0 conn(0x7fa830097360 msgr2=0x7fa830139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:46.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa82ffff700 1 --2- 172.21.15.7:0/3751390207 >> v2:172.21.15.50:3300/0 conn(0x7fa830097360 0x7fa830139720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:46.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa82ffff700 1 -- 172.21.15.7:0/3751390207 >> v1:172.21.15.50:6789/0 conn(0x7fa830096a50 legacy=0x7fa8301327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:46.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa82ffff700 10 monclient: _finish_hunting 0 2023-09-30T01:57:46.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa82ffff700 1 monclient: found mon.noname-a 2023-09-30T01:57:46.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa82ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:46.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa82ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:57:46.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa82ffff700 1 -- 172.21.15.7:0/3751390207 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa830099480 con 0x7fa8300a2390 2023-09-30T01:57:46.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.142+0000 7fa8355b4700 1 --2- 172.21.15.7:0/3751390207 >> v2:172.21.15.50:3300/0 conn(0x7fa830097360 0x7fa830139720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:57:46.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa82ffff700 1 -- 172.21.15.7:0/3751390207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8300a2390 msgr2=0x7fa830096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:46.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa82ffff700 1 --2- 172.21.15.7:0/3751390207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8300a2390 0x7fa830096e20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa830098850 tx=0x7fa820009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e990f699b21b9696 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:46.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa82f7fe700 1 -- 172.21.15.7:0/3751390207 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82000e070 con 0x7fa8300a2390 2023-09-30T01:57:46.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa82f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:46.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa82f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:57:46.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa82f7fe700 10 monclient: dump: 2023-09-30T01:57:46.150 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:46.151 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:46.151 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:46.151 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:46.151 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:46.151 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:46.151 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:46.151 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:46.151 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:46.151 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:46.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa82f7fe700 10 monclient: _finish_auth 0 2023-09-30T01:57:46.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa82f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:46.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa82f7fe700 1 -- 172.21.15.7:0/3751390207 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82000bc20 con 0x7fa8300a2390 2023-09-30T01:57:46.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa837017700 5 monclient: authenticate success, global_id 4548 2023-09-30T01:57:46.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa837017700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:57:46.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa82f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:46.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa82f7fe700 1 -- 172.21.15.7:0/3751390207 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8200092e0 con 0x7fa8300a2390 2023-09-30T01:57:46.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa837017700 10 monclient: get_monmap_and_config success 2023-09-30T01:57:46.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa837017700 4 set_mon_vals no callback set 2023-09-30T01:57:46.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa82f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:46.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa82f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:57:46.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa82f7fe700 10 monclient: dump: 2023-09-30T01:57:46.152 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:46.152 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:46.153 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:46.153 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:46.153 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:46.153 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:46.153 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:46.153 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:46.153 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:46.153 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:46.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa837017700 10 monclient: shutdown 2023-09-30T01:57:46.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa837017700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:46.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa837017700 1 -- 172.21.15.7:0/3751390207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8300a2390 msgr2=0x7fa830096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:46.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa837017700 1 --2- 172.21.15.7:0/3751390207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8300a2390 0x7fa830096e20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa830098850 tx=0x7fa820009840 comp rx=0 tx=0).stop 2023-09-30T01:57:46.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa837017700 1 -- 172.21.15.7:0/3751390207 shutdown_connections 2023-09-30T01:57:46.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa837017700 1 --2- 172.21.15.7:0/3751390207 >> v2:172.21.15.50:3300/0 conn(0x7fa830097360 0x7fa830139720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:46.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa837017700 1 --2- 172.21.15.7:0/3751390207 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa8300a2390 0x7fa830096e20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:46.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa837017700 1 -- 172.21.15.7:0/3751390207 >> 172.21.15.7:0/3751390207 conn(0x7fa830070cf0 msgr2=0x7fa8300710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:46.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa837017700 1 -- 172.21.15.7:0/3751390207 shutdown_connections 2023-09-30T01:57:46.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.145+0000 7fa837017700 1 -- 172.21.15.7:0/3751390207 wait complete. 2023-09-30T01:57:46.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.146+0000 7fa837017700 10 monclient: build_initial_monmap 2023-09-30T01:57:46.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.146+0000 7fa837017700 10 monclient: monmap: 2023-09-30T01:57:46.155 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:46.155 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:57:46.155 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:57:46.155 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:57:46.155 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:46.155 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:46.155 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:57:46.155 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:57:46.155 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:57:46.155 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:46.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.146+0000 7fa837017700 1 Processor -- start 2023-09-30T01:57:46.159 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa837017700 1 -- start start 2023-09-30T01:57:46.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa837017700 10 monclient: init 2023-09-30T01:57:46.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa837017700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:46.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa837017700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:57:46.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa837017700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa830096a50 0x7fa83012f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:46.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa837017700 10 monclient(hunting): picked mon.noname-a con 0x7fa830096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:57:46.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa837017700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa830097360 0x7fa83012f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:46.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa837017700 10 monclient(hunting): picked mon.noname-c con 0x7fa830097360 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:46.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa837017700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8300a2390 0x7fa83009bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:46.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa837017700 10 monclient(hunting): picked mon.noname-b con 0x7fa8300a2390 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:57:46.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa837017700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:46.160 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa837017700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8300a0360 con 0x7fa830096a50 2023-09-30T01:57:46.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa837017700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:46.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa837017700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa83009f790 con 0x7fa8300a2390 2023-09-30T01:57:46.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa837017700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:46.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa837017700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa83009f4b0 con 0x7fa830097360 2023-09-30T01:57:46.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa837017700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:46.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa837017700 10 monclient(hunting): authenticate will time out at 1355.212558s 2023-09-30T01:57:46.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa834db3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa830096a50 0x7fa83012f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:46.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa834db3700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa830096a50 0x7fa83012f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52308/0 (socket says 172.21.15.7:52308) 2023-09-30T01:57:46.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa834db3700 1 -- 172.21.15.7:0/3710570100 learned_addr learned my addr 172.21.15.7:0/3710570100 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:46.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa834db3700 10 monclient(hunting): get_auth_request con 0x7fa830096a50 auth_method 0 2023-09-30T01:57:46.161 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa834db3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:46.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa834db3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:46.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa834db3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:46.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa82ffff700 1 --2- 172.21.15.7:0/3710570100 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa830097360 0x7fa83012f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:46.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa82ffff700 10 monclient(hunting): get_auth_request con 0x7fa830097360 auth_method 0 2023-09-30T01:57:46.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa82ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:46.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa82ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:46.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa82ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:46.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa834db3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:46.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa834db3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:46.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa834db3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:46.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa82ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:46.162 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa82ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:46.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa82ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:46.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa8355b4700 1 --2- 172.21.15.7:0/3710570100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8300a2390 0x7fa83009bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:46.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa8355b4700 10 monclient(hunting): get_auth_request con 0x7fa8300a2390 auth_method 0 2023-09-30T01:57:46.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa8355b4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:46.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa8355b4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:46.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa8355b4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:46.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa834db3700 10 monclient(hunting): handle_auth_done global_id 4551 payload 824 2023-09-30T01:57:46.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa834db3700 1 -- 172.21.15.7:0/3710570100 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa830097360 msgr2=0x7fa83012f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:46.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa834db3700 1 --2- 172.21.15.7:0/3710570100 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa830097360 0x7fa83012f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:46.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa834db3700 1 -- 172.21.15.7:0/3710570100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8300a2390 msgr2=0x7fa83009bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:46.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa834db3700 1 --2- 172.21.15.7:0/3710570100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8300a2390 0x7fa83009bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:46.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa834db3700 10 monclient: _finish_hunting 0 2023-09-30T01:57:46.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa834db3700 1 monclient: found mon.noname-a 2023-09-30T01:57:46.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa834db3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:46.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa834db3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:57:46.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.147+0000 7fa834db3700 1 -- 172.21.15.7:0/3710570100 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa820009ae0 con 0x7fa830096a50 2023-09-30T01:57:46.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa834db3700 1 -- 172.21.15.7:0/3710570100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa830096a50 msgr2=0x7fa83012f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:46.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa834db3700 1 --2- 172.21.15.7:0/3710570100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa830096a50 0x7fa83012f150 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa81800c600 tx=0x7fa818007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:46.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa82cff9700 1 -- 172.21.15.7:0/3710570100 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8180117a0 con 0x7fa830096a50 2023-09-30T01:57:46.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa82cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:46.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa82cff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:57:46.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa82cff9700 10 monclient: dump: 2023-09-30T01:57:46.165 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:46.165 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:46.165 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:46.165 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:46.165 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:46.166 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:46.166 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:46.166 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:46.166 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:46.166 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:46.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa82cff9700 10 monclient: _finish_auth 0 2023-09-30T01:57:46.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa82cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:46.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa837017700 5 monclient: authenticate success, global_id 4551 2023-09-30T01:57:46.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa82cff9700 1 -- 172.21.15.7:0/3710570100 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa818011900 con 0x7fa830096a50 2023-09-30T01:57:46.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa82cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:46.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa82cff9700 1 -- 172.21.15.7:0/3710570100 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa818011cc0 con 0x7fa830096a50 2023-09-30T01:57:46.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa837017700 10 monclient: _renew_subs 2023-09-30T01:57:46.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa837017700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:46.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa837017700 1 -- 172.21.15.7:0/3710570100 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa83009e340 con 0x7fa830096a50 2023-09-30T01:57:46.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa82effd700 4 set_mon_vals no callback set 2023-09-30T01:57:46.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa82cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:46.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa82cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:57:46.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa82cff9700 10 monclient: dump: 2023-09-30T01:57:46.167 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:46.167 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:46.167 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:46.167 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:46.167 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:46.168 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:46.168 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:46.168 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:46.168 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:46.168 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:46.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa837017700 10 monclient: _renew_subs 2023-09-30T01:57:46.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa837017700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:46.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.150+0000 7fa837017700 1 -- 172.21.15.7:0/3710570100 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa83009d720 con 0x7fa830096a50 2023-09-30T01:57:46.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.151+0000 7fa82cff9700 1 -- 172.21.15.7:0/3710570100 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fa818012630 con 0x7fa830096a50 2023-09-30T01:57:46.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.151+0000 7fa837017700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:46.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.151+0000 7fa837017700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:46.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.151+0000 7fa837017700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:46.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.151+0000 7fa82cff9700 1 --2- 172.21.15.7:0/3710570100 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa8100357e0 0x7fa810037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:46.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.151+0000 7fa837017700 1 -- 172.21.15.7:0/3710570100 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8300a4910 con 0x7fa830096a50 2023-09-30T01:57:46.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.151+0000 7fa82cff9700 1 -- 172.21.15.7:0/3710570100 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fa818047a80 con 0x7fa830096a50 2023-09-30T01:57:46.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.152+0000 7fa82ffff700 1 --2- 172.21.15.7:0/3710570100 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa8100357e0 0x7fa810037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:46.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.152+0000 7fa82ffff700 10 monclient: get_auth_request con 0x7fa8100357e0 auth_method 0 2023-09-30T01:57:46.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.154+0000 7fa82ffff700 1 --2- 172.21.15.7:0/3710570100 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa8100357e0 0x7fa810037c90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa820009a60 tx=0x7fa820017000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:46.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.156+0000 7fa82cff9700 1 -- 172.21.15.7:0/3710570100 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa81801c460 con 0x7fa830096a50 2023-09-30T01:57:46.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.156+0000 7fa82cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:46.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.156+0000 7fa82cff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:57:46.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.395+0000 7fa837017700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:57:46.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.395+0000 7fa837017700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:46.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.395+0000 7fa837017700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:46.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.395+0000 7fa837017700 1 -- 172.21.15.7:0/3710570100 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa8300a4bf0 con 0x7fa830096a50 2023-09-30T01:57:46.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.395+0000 7fa82cff9700 1 -- 172.21.15.7:0/3710570100 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa8180102e0 con 0x7fa830096a50 2023-09-30T01:57:46.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.395+0000 7fa82cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:46.396 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.395+0000 7fa82cff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:57:46.397 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:57:46.397 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:57:46.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.397+0000 7fa837017700 1 -- 172.21.15.7:0/3710570100 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa8100357e0 msgr2=0x7fa810037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:46.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.397+0000 7fa837017700 1 --2- 172.21.15.7:0/3710570100 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa8100357e0 0x7fa810037c90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa820009a60 tx=0x7fa820017000 comp rx=0 tx=0).stop 2023-09-30T01:57:46.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.397+0000 7fa837017700 10 monclient: shutdown 2023-09-30T01:57:46.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.397+0000 7fa837017700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:46.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.397+0000 7fa837017700 1 -- 172.21.15.7:0/3710570100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa830096a50 msgr2=0x7fa83012f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:46.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.397+0000 7fa837017700 1 --2- 172.21.15.7:0/3710570100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa830096a50 0x7fa83012f150 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa81800c600 tx=0x7fa818007f60 comp rx=0 tx=0).stop 2023-09-30T01:57:46.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.397+0000 7fa837017700 1 -- 172.21.15.7:0/3710570100 shutdown_connections 2023-09-30T01:57:46.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.397+0000 7fa837017700 1 --2- 172.21.15.7:0/3710570100 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa8100357e0 0x7fa810037c90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:46.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.397+0000 7fa837017700 1 --2- 172.21.15.7:0/3710570100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa830096a50 0x7fa83012f150 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:46.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.397+0000 7fa837017700 1 --2- 172.21.15.7:0/3710570100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa8300a2390 0x7fa83009bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:46.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.397+0000 7fa837017700 1 --2- 172.21.15.7:0/3710570100 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa830097360 0x7fa83012f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:46.399 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.397+0000 7fa837017700 1 -- 172.21.15.7:0/3710570100 >> 172.21.15.7:0/3710570100 conn(0x7fa830070cf0 msgr2=0x7fa830130800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:46.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.398+0000 7fa837017700 1 -- 172.21.15.7:0/3710570100 shutdown_connections 2023-09-30T01:57:46.400 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:46.398+0000 7fa837017700 1 -- 172.21.15.7:0/3710570100 wait complete. 2023-09-30T01:57:49.410 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:57:49.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.540+0000 7f070d175700 10 monclient: get_monmap_and_config 2023-09-30T01:57:49.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.540+0000 7f070d175700 10 monclient: build_initial_monmap 2023-09-30T01:57:49.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.540+0000 7f070d175700 10 monclient: monmap: 2023-09-30T01:57:49.540 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:49.540 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:49.541 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:57:49.541402+0000 2023-09-30T01:57:49.541 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:57:49.541402+0000 2023-09-30T01:57:49.541 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:49.541 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:49.541 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:57:49.541 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:57:49.541 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:57:49.541 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:57:49.541 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:57:49.541 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:49.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.540+0000 7f070d175700 1 Processor -- start 2023-09-30T01:57:49.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.540+0000 7f070d175700 1 -- start start 2023-09-30T01:57:49.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.540+0000 7f070d175700 10 monclient: init 2023-09-30T01:57:49.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070d175700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:49.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070d175700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2023-09-30T01:57:49.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070d175700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0708096a50 addr v1:172.21.15.50:6789/0 2023-09-30T01:57:49.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070d175700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f07080a2390 0x7f0708096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:49.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070d175700 10 monclient(hunting): picked mon.noname-b con 0x7f07080a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:49.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070d175700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0708097360 addr v1:172.21.15.7:6789/0 2023-09-30T01:57:49.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070d175700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07080980e0 con 0x7f0708097360 2023-09-30T01:57:49.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070d175700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0708132eb0 con 0x7f0708096a50 2023-09-30T01:57:49.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070d175700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:49.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070d175700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07080991a0 con 0x7f07080a2390 2023-09-30T01:57:49.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070d175700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:49.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070d175700 10 monclient(hunting): authenticate will time out at 1358.606467s 2023-09-30T01:57:49.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070659c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f07080a2390 0x7f0708096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:49.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070759e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0708097360 0x7f0708139720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41840/0 (socket says 172.21.15.7:41840) 2023-09-30T01:57:49.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070759e700 1 -- 172.21.15.7:0/2341969228 learned_addr learned my addr 172.21.15.7:0/2341969228 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:57:49.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070659c700 10 monclient(hunting): get_auth_request con 0x7f07080a2390 auth_method 0 2023-09-30T01:57:49.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:49.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:49.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:49.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:49.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:49.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:49.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070659c700 10 monclient(hunting): handle_auth_done global_id 4568 payload 274 2023-09-30T01:57:49.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070659c700 1 -- 172.21.15.7:0/2341969228 >> v1:172.21.15.50:6789/0 conn(0x7f0708096a50 legacy=0x7f07081327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:49.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070659c700 1 -- 172.21.15.7:0/2341969228 >> v1:172.21.15.7:6789/0 conn(0x7f0708097360 legacy=0x7f0708139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:49.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070659c700 10 monclient: _finish_hunting 0 2023-09-30T01:57:49.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070659c700 1 monclient: found mon.noname-b 2023-09-30T01:57:49.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:49.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070659c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:57:49.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.541+0000 7f070659c700 1 -- 172.21.15.7:0/2341969228 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0708099480 con 0x7f07080a2390 2023-09-30T01:57:49.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f070659c700 1 -- 172.21.15.7:0/2341969228 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f07080a2390 msgr2=0x7f0708096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:49.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f070659c700 1 --2- 172.21.15.7:0/2341969228 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f07080a2390 0x7f0708096e20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f06fc009910 tx=0x7f06fc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5281b45e38f7b3f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:49.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f0705d9b700 1 -- 172.21.15.7:0/2341969228 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc00b9f0 con 0x7f07080a2390 2023-09-30T01:57:49.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f0705d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:49.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f0705d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:57:49.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f0705d9b700 10 monclient: dump: 2023-09-30T01:57:49.558 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:49.558 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:49.558 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:49.558 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:49.558 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:49.558 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:49.558 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:49.559 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:49.559 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:49.559 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:49.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f0705d9b700 10 monclient: _finish_auth 0 2023-09-30T01:57:49.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f0705d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:49.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f0705d9b700 1 -- 172.21.15.7:0/2341969228 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06fc00bb50 con 0x7f07080a2390 2023-09-30T01:57:49.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f070d175700 5 monclient: authenticate success, global_id 4568 2023-09-30T01:57:49.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f070d175700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:57:49.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f0705d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:49.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f070d175700 10 monclient: get_monmap_and_config success 2023-09-30T01:57:49.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f0705d9b700 1 -- 172.21.15.7:0/2341969228 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc011420 con 0x7f07080a2390 2023-09-30T01:57:49.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f0705d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:49.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f070d175700 4 set_mon_vals no callback set 2023-09-30T01:57:49.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f0705d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:57:49.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f0705d9b700 10 monclient: dump: 2023-09-30T01:57:49.561 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:49.561 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:49.561 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:49.562 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:49.562 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:49.562 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:49.562 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:49.562 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:49.562 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:49.562 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:49.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f070d175700 10 monclient: shutdown 2023-09-30T01:57:49.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f070d175700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:49.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f070d175700 1 -- 172.21.15.7:0/2341969228 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f07080a2390 msgr2=0x7f0708096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:49.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f070d175700 1 --2- 172.21.15.7:0/2341969228 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f07080a2390 0x7f0708096e20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f06fc009910 tx=0x7f06fc0091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:49.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f070d175700 1 -- 172.21.15.7:0/2341969228 shutdown_connections 2023-09-30T01:57:49.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f070d175700 1 --2- 172.21.15.7:0/2341969228 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f07080a2390 0x7f0708096e20 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:49.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f070d175700 1 -- 172.21.15.7:0/2341969228 >> 172.21.15.7:0/2341969228 conn(0x7f0708070cf0 msgr2=0x7f07080710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:49.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f070d175700 1 -- 172.21.15.7:0/2341969228 shutdown_connections 2023-09-30T01:57:49.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.544+0000 7f070d175700 1 -- 172.21.15.7:0/2341969228 wait complete. 2023-09-30T01:57:49.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.545+0000 7f070d175700 10 monclient: build_initial_monmap 2023-09-30T01:57:49.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.545+0000 7f070d175700 10 monclient: monmap: 2023-09-30T01:57:49.564 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:49.564 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:57:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:57:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:57:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:57:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:57:49.565 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:57:49.565 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.545+0000 7f070d175700 1 Processor -- start 2023-09-30T01:57:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.545+0000 7f070d175700 1 -- start start 2023-09-30T01:57:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.545+0000 7f070d175700 10 monclient: init 2023-09-30T01:57:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070d175700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070d175700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:57:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070d175700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0708096a50 0x7f070809b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:49.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070d175700 10 monclient(hunting): picked mon.noname-a con 0x7f0708096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:57:49.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070d175700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0708097360 0x7f070809bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:49.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070d175700 10 monclient(hunting): picked mon.noname-c con 0x7f0708097360 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:49.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070d175700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07080a2390 0x7f070809c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:49.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070d175700 10 monclient(hunting): picked mon.noname-b con 0x7f07080a2390 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:57:49.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070d175700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:49.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070d175700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070813aa90 con 0x7f0708096a50 2023-09-30T01:57:49.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070d175700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:49.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070d175700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070809f0e0 con 0x7f07080a2390 2023-09-30T01:57:49.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070d175700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:49.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070d175700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f070809ee00 con 0x7f0708097360 2023-09-30T01:57:49.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070d175700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:49.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070d175700 10 monclient(hunting): authenticate will time out at 1358.611380s 2023-09-30T01:57:49.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f0706d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0708096a50 0x7f070809b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:49.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070659c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0708097360 0x7f070809bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070659c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0708097360 0x7f070809bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:35650/0 (socket says 172.21.15.7:35650) 2023-09-30T01:57:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070659c700 1 -- 172.21.15.7:0/2300753735 learned_addr learned my addr 172.21.15.7:0/2300753735 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070659c700 10 monclient(hunting): get_auth_request con 0x7f0708097360 auth_method 0 2023-09-30T01:57:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f0706d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0708096a50 0x7f070809b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52324/0 (socket says 172.21.15.7:52324) 2023-09-30T01:57:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:49.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f0706d9d700 10 monclient(hunting): get_auth_request con 0x7f0708096a50 auth_method 0 2023-09-30T01:57:49.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f0706d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:49.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f0706d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:49.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f0706d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:49.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:49.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f0706d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f0706d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f0706d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070759e700 1 --2- 172.21.15.7:0/2300753735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07080a2390 0x7f070809c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070659c700 10 monclient(hunting): handle_auth_done global_id 4571 payload 824 2023-09-30T01:57:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070659c700 1 -- 172.21.15.7:0/2300753735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07080a2390 msgr2=0x7f070809c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:49.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070659c700 1 --2- 172.21.15.7:0/2300753735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07080a2390 0x7f070809c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:49.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070659c700 1 -- 172.21.15.7:0/2300753735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0708096a50 msgr2=0x7f070809b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:49.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070659c700 1 --2- 172.21.15.7:0/2300753735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0708096a50 0x7f070809b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:49.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070659c700 10 monclient: _finish_hunting 0 2023-09-30T01:57:49.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070659c700 1 monclient: found mon.noname-c 2023-09-30T01:57:49.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:49.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070659c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:57:49.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070659c700 1 -- 172.21.15.7:0/2300753735 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06fc009420 con 0x7f0708097360 2023-09-30T01:57:49.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070759e700 10 monclient: get_auth_request con 0x7f07080a2390 auth_method 0 2023-09-30T01:57:49.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f0706d9d700 1 --2- 172.21.15.7:0/2300753735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0708096a50 0x7f070809b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:49.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.546+0000 7f070759e700 1 --2- 172.21.15.7:0/2300753735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07080a2390 0x7f070809c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:57:49.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f070659c700 1 -- 172.21.15.7:0/2300753735 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0708097360 msgr2=0x7f070809bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:49.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f070659c700 1 --2- 172.21.15.7:0/2300753735 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0708097360 0x7f070809bec0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f06fc002000 tx=0x7f06fc016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:49.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f06eeffd700 1 -- 172.21.15.7:0/2300753735 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc006b30 con 0x7f0708097360 2023-09-30T01:57:49.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f06eeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:49.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f06eeffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:57:49.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f06eeffd700 10 monclient: dump: 2023-09-30T01:57:49.574 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:49.574 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:49.574 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:49.574 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:49.574 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:49.574 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:49.575 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:49.575 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:49.575 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:49.575 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:49.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f06eeffd700 10 monclient: _finish_auth 0 2023-09-30T01:57:49.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f06eeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:49.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f070d175700 5 monclient: authenticate success, global_id 4571 2023-09-30T01:57:49.575 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f06eeffd700 1 -- 172.21.15.7:0/2300753735 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06fc00bed0 con 0x7f0708097360 2023-09-30T01:57:49.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f06eeffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:49.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f06eeffd700 1 -- 172.21.15.7:0/2300753735 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc01c8c0 con 0x7f0708097360 2023-09-30T01:57:49.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f06eeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:49.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f06eeffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:57:49.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f06eeffd700 10 monclient: dump: 2023-09-30T01:57:49.576 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:49.576 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:49.576 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:49.577 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:49.577 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:49.577 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:49.577 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:49.577 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:49.577 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:49.577 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:49.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f070d175700 10 monclient: _renew_subs 2023-09-30T01:57:49.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f070d175700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:49.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f070d175700 1 -- 172.21.15.7:0/2300753735 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f070809f3c0 con 0x7f0708097360 2023-09-30T01:57:49.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f070559a700 4 set_mon_vals no callback set 2023-09-30T01:57:49.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f070d175700 10 monclient: _renew_subs 2023-09-30T01:57:49.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f070d175700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:49.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.549+0000 7f070d175700 1 -- 172.21.15.7:0/2300753735 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f070809df10 con 0x7f0708097360 2023-09-30T01:57:49.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.550+0000 7f06eeffd700 1 -- 172.21.15.7:0/2300753735 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f06fc01cc70 con 0x7f0708097360 2023-09-30T01:57:49.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.550+0000 7f070d175700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:49.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.550+0000 7f070d175700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:49.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.550+0000 7f070d175700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:49.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.550+0000 7f06eeffd700 1 --2- 172.21.15.7:0/2300753735 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f06f40357e0 0x7f06f4037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:49.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.550+0000 7f070d175700 1 -- 172.21.15.7:0/2300753735 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f070809df10 con 0x7f0708097360 2023-09-30T01:57:49.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.550+0000 7f06eeffd700 1 -- 172.21.15.7:0/2300753735 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f06fc04a1b0 con 0x7f0708097360 2023-09-30T01:57:49.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.550+0000 7f0706d9d700 1 --2- 172.21.15.7:0/2300753735 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f06f40357e0 0x7f06f4037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:49.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.550+0000 7f0706d9d700 10 monclient: get_auth_request con 0x7f06f40357e0 auth_method 0 2023-09-30T01:57:49.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.553+0000 7f0706d9d700 1 --2- 172.21.15.7:0/2300753735 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f06f40357e0 0x7f06f4037c90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f06f8002e00 tx=0x7f06f8007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:49.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.555+0000 7f06eeffd700 1 -- 172.21.15.7:0/2300753735 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f06fc0237a0 con 0x7f0708097360 2023-09-30T01:57:49.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.555+0000 7f06eeffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:49.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.555+0000 7f06eeffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:57:49.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.789+0000 7f070d175700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:57:49.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.789+0000 7f070d175700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:49.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.789+0000 7f070d175700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:49.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.789+0000 7f070d175700 1 -- 172.21.15.7:0/2300753735 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f070809df10 con 0x7f0708097360 2023-09-30T01:57:49.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.789+0000 7f06eeffd700 1 -- 172.21.15.7:0/2300753735 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f06fc02c340 con 0x7f0708097360 2023-09-30T01:57:49.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.789+0000 7f06eeffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:49.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.789+0000 7f06eeffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:57:49.790 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:57:49.790 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:57:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.792+0000 7f070d175700 1 -- 172.21.15.7:0/2300753735 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f06f40357e0 msgr2=0x7f06f4037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.792+0000 7f070d175700 1 --2- 172.21.15.7:0/2300753735 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f06f40357e0 0x7f06f4037c90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f06f8002e00 tx=0x7f06f8007ea0 comp rx=0 tx=0).stop 2023-09-30T01:57:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.792+0000 7f070d175700 10 monclient: shutdown 2023-09-30T01:57:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.792+0000 7f070d175700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.792+0000 7f070d175700 1 -- 172.21.15.7:0/2300753735 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0708097360 msgr2=0x7f070809bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.792+0000 7f070d175700 1 --2- 172.21.15.7:0/2300753735 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0708097360 0x7f070809bec0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f06fc002000 tx=0x7f06fc016000 comp rx=0 tx=0).stop 2023-09-30T01:57:49.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.792+0000 7f070d175700 1 -- 172.21.15.7:0/2300753735 shutdown_connections 2023-09-30T01:57:49.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.792+0000 7f070d175700 1 --2- 172.21.15.7:0/2300753735 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f06f40357e0 0x7f06f4037c90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:49.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.792+0000 7f070d175700 1 --2- 172.21.15.7:0/2300753735 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0708096a50 0x7f070809b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:49.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.792+0000 7f070d175700 1 --2- 172.21.15.7:0/2300753735 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07080a2390 0x7f070809c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:49.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.792+0000 7f070d175700 1 --2- 172.21.15.7:0/2300753735 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0708097360 0x7f070809bec0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:49.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.792+0000 7f070d175700 1 -- 172.21.15.7:0/2300753735 >> 172.21.15.7:0/2300753735 conn(0x7f0708070cf0 msgr2=0x7f07080a3670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:49.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.792+0000 7f070d175700 1 -- 172.21.15.7:0/2300753735 shutdown_connections 2023-09-30T01:57:49.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:49.792+0000 7f070d175700 1 -- 172.21.15.7:0/2300753735 wait complete. 2023-09-30T01:57:52.804 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:57:52.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.935+0000 7fabf1d7b700 10 monclient: get_monmap_and_config 2023-09-30T01:57:52.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.935+0000 7fabf1d7b700 10 monclient: build_initial_monmap 2023-09-30T01:57:52.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.935+0000 7fabf1d7b700 10 monclient: monmap: 2023-09-30T01:57:52.935 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:52.935 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:52.935 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:57:52.936126+0000 2023-09-30T01:57:52.936 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:57:52.936126+0000 2023-09-30T01:57:52.936 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:52.936 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:52.936 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:57:52.936 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:57:52.936 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:57:52.936 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:57:52.936 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:57:52.936 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:52.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.935+0000 7fabf1d7b700 1 Processor -- start 2023-09-30T01:57:52.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.935+0000 7fabf1d7b700 1 -- start start 2023-09-30T01:57:52.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.935+0000 7fabf1d7b700 10 monclient: init 2023-09-30T01:57:52.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabf1d7b700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:52.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabf1d7b700 10 monclient: _add_conns ranks=[0,3,4,1,2] 2023-09-30T01:57:52.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabf1d7b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fabec096a50 addr v1:172.21.15.7:6789/0 2023-09-30T01:57:52.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabf1d7b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fabec096e20 0x7fabec097220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabf1d7b700 10 monclient(hunting): picked mon.noname-a con 0x7fabec096e20 addr v2:172.21.15.7:3300/0 2023-09-30T01:57:52.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabf1d7b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fabec137710 0x7fabec139af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabf1d7b700 10 monclient(hunting): picked mon.noname-c con 0x7fabec137710 addr v2:172.21.15.50:3300/0 2023-09-30T01:57:52.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabf1d7b700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabec132d90 con 0x7fabec096a50 2023-09-30T01:57:52.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabf1d7b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:52.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabf1d7b700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabec097b40 con 0x7fabec096e20 2023-09-30T01:57:52.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabf1d7b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:52.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabf1d7b700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabec099160 con 0x7fabec137710 2023-09-30T01:57:52.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabf1d7b700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:52.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabf1d7b700 10 monclient(hunting): authenticate will time out at 1362.001433s 2023-09-30T01:57:52.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabeaffd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fabec096e20 0x7fabec097220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabeb7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fabec096a50 0x7fabec132680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41842/0 (socket says 172.21.15.7:41842) 2023-09-30T01:57:52.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabeb7fe700 1 -- 172.21.15.7:0/2997478863 learned_addr learned my addr 172.21.15.7:0/2997478863 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:57:52.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabeaffd700 10 monclient(hunting): get_auth_request con 0x7fabec096e20 auth_method 0 2023-09-30T01:57:52.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabeaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:52.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabeaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:52.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabeaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:52.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabeaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:52.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabeaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:52.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabeaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:52.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabebfff700 1 --2- 172.21.15.7:0/2997478863 >> v2:172.21.15.50:3300/0 conn(0x7fabec137710 0x7fabec139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabebfff700 10 monclient(hunting): get_auth_request con 0x7fabec137710 auth_method 0 2023-09-30T01:57:52.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:52.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabebfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:52.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabebfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:52.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:52.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:52.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:52.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabeaffd700 10 monclient(hunting): handle_auth_done global_id 4557 payload 274 2023-09-30T01:57:52.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabeaffd700 1 -- 172.21.15.7:0/2997478863 >> v2:172.21.15.50:3300/0 conn(0x7fabec137710 msgr2=0x7fabec139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:52.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabeaffd700 1 --2- 172.21.15.7:0/2997478863 >> v2:172.21.15.50:3300/0 conn(0x7fabec137710 0x7fabec139af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabeaffd700 1 -- 172.21.15.7:0/2997478863 >> v1:172.21.15.7:6789/0 conn(0x7fabec096a50 legacy=0x7fabec132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:52.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabeaffd700 10 monclient: _finish_hunting 0 2023-09-30T01:57:52.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabeaffd700 1 monclient: found mon.noname-a 2023-09-30T01:57:52.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabeaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:52.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabeaffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:57:52.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.936+0000 7fabeaffd700 1 -- 172.21.15.7:0/2997478863 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabec099720 con 0x7fabec096e20 2023-09-30T01:57:52.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabeaffd700 1 -- 172.21.15.7:0/2997478863 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabec096e20 msgr2=0x7fabec097220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:52.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabeaffd700 1 --2- 172.21.15.7:0/2997478863 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabec096e20 0x7fabec097220 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fabec099470 tx=0x7fabe00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a0036233fc43d7f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabea7fc700 1 -- 172.21.15.7:0/2997478863 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe0011840 con 0x7fabec096e20 2023-09-30T01:57:52.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:52.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabea7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:57:52.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabea7fc700 10 monclient: dump: 2023-09-30T01:57:52.944 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:52.944 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:52.944 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:52.945 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:52.945 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:52.945 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:52.945 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:52.945 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:52.945 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:52.945 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:52.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabea7fc700 10 monclient: _finish_auth 0 2023-09-30T01:57:52.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabea7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:52.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabea7fc700 1 -- 172.21.15.7:0/2997478863 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe00119a0 con 0x7fabec096e20 2023-09-30T01:57:52.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabea7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:52.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabea7fc700 1 -- 172.21.15.7:0/2997478863 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe0011d40 con 0x7fabec096e20 2023-09-30T01:57:52.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabf1d7b700 5 monclient: authenticate success, global_id 4557 2023-09-30T01:57:52.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabf1d7b700 10 monclient: get_monmap_and_config success 2023-09-30T01:57:52.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:52.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabf1d7b700 4 set_mon_vals no callback set 2023-09-30T01:57:52.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabea7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:57:52.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabea7fc700 10 monclient: dump: 2023-09-30T01:57:52.946 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:52.946 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:52.946 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:52.946 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:52.946 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:52.946 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:52.947 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:52.947 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:52.947 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:52.947 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:52.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabf1d7b700 10 monclient: shutdown 2023-09-30T01:57:52.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabf1d7b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:52.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabf1d7b700 1 -- 172.21.15.7:0/2997478863 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabec096e20 msgr2=0x7fabec097220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabf1d7b700 1 --2- 172.21.15.7:0/2997478863 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabec096e20 0x7fabec097220 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fabec099470 tx=0x7fabe00091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabf1d7b700 1 -- 172.21.15.7:0/2997478863 shutdown_connections 2023-09-30T01:57:52.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabf1d7b700 1 --2- 172.21.15.7:0/2997478863 >> v2:172.21.15.50:3300/0 conn(0x7fabec137710 0x7fabec139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabf1d7b700 1 --2- 172.21.15.7:0/2997478863 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabec096e20 0x7fabec097220 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabf1d7b700 1 -- 172.21.15.7:0/2997478863 >> 172.21.15.7:0/2997478863 conn(0x7fabec070cf0 msgr2=0x7fabec0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:52.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabf1d7b700 1 -- 172.21.15.7:0/2997478863 shutdown_connections 2023-09-30T01:57:52.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.939+0000 7fabf1d7b700 1 -- 172.21.15.7:0/2997478863 wait complete. 2023-09-30T01:57:52.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.940+0000 7fabf1d7b700 10 monclient: build_initial_monmap 2023-09-30T01:57:52.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.940+0000 7fabf1d7b700 10 monclient: monmap: 2023-09-30T01:57:52.948 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:52.948 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:57:52.948 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:57:52.948 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:57:52.948 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:52.948 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:52.948 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:57:52.948 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:57:52.948 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:57:52.949 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:52.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.940+0000 7fabf1d7b700 1 Processor -- start 2023-09-30T01:57:52.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabf1d7b700 1 -- start start 2023-09-30T01:57:52.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabf1d7b700 10 monclient: init 2023-09-30T01:57:52.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabf1d7b700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:52.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabf1d7b700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:57:52.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabf1d7b700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fabec096a50 0x7fabec0a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabf1d7b700 10 monclient(hunting): picked mon.noname-c con 0x7fabec096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:52.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabf1d7b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabec096e20 0x7fabec09cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabf1d7b700 10 monclient(hunting): picked mon.noname-b con 0x7fabec096e20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:57:52.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabf1d7b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabec137710 0x7fabec09d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabf1d7b700 10 monclient(hunting): picked mon.noname-a con 0x7fabec137710 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:57:52.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabf1d7b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:52.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabf1d7b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabec0a16b0 con 0x7fabec137710 2023-09-30T01:57:52.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabf1d7b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:52.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabf1d7b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabec09ecb0 con 0x7fabec096e20 2023-09-30T01:57:52.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabf1d7b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:52.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabf1d7b700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabec131db0 con 0x7fabec096a50 2023-09-30T01:57:52.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabf1d7b700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:52.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabf1d7b700 10 monclient(hunting): authenticate will time out at 1362.006468s 2023-09-30T01:57:52.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabebfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabec137710 0x7fabec09d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabeb7fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fabec096a50 0x7fabec0a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabebfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabec137710 0x7fabec09d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52350/0 (socket says 172.21.15.7:52350) 2023-09-30T01:57:52.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabebfff700 1 -- 172.21.15.7:0/10896953 learned_addr learned my addr 172.21.15.7:0/10896953 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:52.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabeb7fe700 10 monclient(hunting): get_auth_request con 0x7fabec096a50 auth_method 0 2023-09-30T01:57:52.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabeb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:52.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabeb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:52.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabeb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:52.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabebfff700 10 monclient(hunting): get_auth_request con 0x7fabec137710 auth_method 0 2023-09-30T01:57:52.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:52.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabebfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:52.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabebfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:52.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:52.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:52.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabeaffd700 1 --2- 172.21.15.7:0/10896953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabec096e20 0x7fabec09cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:52.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabeb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:52.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabeb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:52.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabeb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:52.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabeaffd700 10 monclient(hunting): get_auth_request con 0x7fabec096e20 auth_method 0 2023-09-30T01:57:52.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabeaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:52.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabeaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:52.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabeaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:52.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabeaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:52.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabeaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:52.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabeaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:52.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabebfff700 10 monclient(hunting): handle_auth_done global_id 4560 payload 824 2023-09-30T01:57:52.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabebfff700 1 -- 172.21.15.7:0/10896953 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fabec096a50 msgr2=0x7fabec0a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabebfff700 1 --2- 172.21.15.7:0/10896953 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fabec096a50 0x7fabec0a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabebfff700 1 -- 172.21.15.7:0/10896953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabec096e20 msgr2=0x7fabec09cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:52.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabebfff700 1 --2- 172.21.15.7:0/10896953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabec096e20 0x7fabec09cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:52.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabebfff700 10 monclient: _finish_hunting 0 2023-09-30T01:57:52.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabebfff700 1 monclient: found mon.noname-a 2023-09-30T01:57:52.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabebfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:52.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabebfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:57:52.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.941+0000 7fabebfff700 1 -- 172.21.15.7:0/10896953 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe0009420 con 0x7fabec137710 2023-09-30T01:57:52.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.942+0000 7fabeb7fe700 1 --2- 172.21.15.7:0/10896953 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fabec096a50 0x7fabec0a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:52.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabebfff700 1 -- 172.21.15.7:0/10896953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabec137710 msgr2=0x7fabec09d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:52.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabebfff700 1 --2- 172.21.15.7:0/10896953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabec137710 0x7fabec09d100 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fabe4009ec0 tx=0x7fabe400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabd37fe700 1 -- 172.21.15.7:0/10896953 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe400cd80 con 0x7fabec137710 2023-09-30T01:57:52.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabd37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:52.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabd37fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:57:52.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabd37fe700 10 monclient: dump: 2023-09-30T01:57:52.960 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:52.960 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:52.960 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:52.960 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:52.960 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:52.960 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:52.960 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:52.960 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:52.960 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:52.960 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:52.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabd37fe700 10 monclient: _finish_auth 0 2023-09-30T01:57:52.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabd37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:52.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabd37fe700 1 -- 172.21.15.7:0/10896953 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe400f040 con 0x7fabec137710 2023-09-30T01:57:52.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabf1d7b700 5 monclient: authenticate success, global_id 4560 2023-09-30T01:57:52.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabf1d7b700 10 monclient: _renew_subs 2023-09-30T01:57:52.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabf1d7b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:52.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabf1d7b700 1 -- 172.21.15.7:0/10896953 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabec09f300 con 0x7fabec137710 2023-09-30T01:57:52.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabd37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:52.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabd37fe700 1 -- 172.21.15.7:0/10896953 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe400cee0 con 0x7fabec137710 2023-09-30T01:57:52.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabd37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:52.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabd37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:57:52.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabd37fe700 10 monclient: dump: 2023-09-30T01:57:52.962 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:52.962 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:52.962 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:52.962 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:52.962 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:52.962 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:52.962 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:52.962 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:52.962 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:52.962 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:52.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabe9ffb700 4 set_mon_vals no callback set 2023-09-30T01:57:52.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabf1d7b700 10 monclient: _renew_subs 2023-09-30T01:57:52.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabf1d7b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:52.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.944+0000 7fabf1d7b700 1 -- 172.21.15.7:0/10896953 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabec0a0270 con 0x7fabec137710 2023-09-30T01:57:52.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.945+0000 7fabd37fe700 1 -- 172.21.15.7:0/10896953 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fabe4014920 con 0x7fabec137710 2023-09-30T01:57:52.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.945+0000 7fabf1d7b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:52.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.945+0000 7fabf1d7b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:52.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.945+0000 7fabf1d7b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:52.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.945+0000 7fabf1d7b700 1 -- 172.21.15.7:0/10896953 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabec064c10 con 0x7fabec137710 2023-09-30T01:57:52.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.945+0000 7fabd37fe700 1 --2- 172.21.15.7:0/10896953 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fabd80357e0 0x7fabd8037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:52.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.945+0000 7fabd37fe700 1 -- 172.21.15.7:0/10896953 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fabe40486c0 con 0x7fabec137710 2023-09-30T01:57:52.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.945+0000 7fabeb7fe700 1 --2- 172.21.15.7:0/10896953 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fabd80357e0 0x7fabd8037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:52.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.945+0000 7fabeb7fe700 10 monclient: get_auth_request con 0x7fabd80357e0 auth_method 0 2023-09-30T01:57:52.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.948+0000 7fabeb7fe700 1 --2- 172.21.15.7:0/10896953 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fabd80357e0 0x7fabd8037c90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fabdc006fd0 tx=0x7fabdc007f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:52.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.950+0000 7fabd37fe700 1 -- 172.21.15.7:0/10896953 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fabe401ad20 con 0x7fabec137710 2023-09-30T01:57:52.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.950+0000 7fabd37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:52.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:52.950+0000 7fabd37fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:57:53.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:53.182+0000 7fabf1d7b700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:57:53.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:53.182+0000 7fabf1d7b700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:53.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:53.182+0000 7fabf1d7b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:53.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:53.182+0000 7fabf1d7b700 1 -- 172.21.15.7:0/10896953 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fabec097760 con 0x7fabec137710 2023-09-30T01:57:53.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:53.183+0000 7fabd37fe700 1 -- 172.21.15.7:0/10896953 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fabe4018080 con 0x7fabec137710 2023-09-30T01:57:53.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:53.183+0000 7fabd37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:53.183 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:53.183+0000 7fabd37fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:57:53.183 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:57:53.184 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:57:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:53.184+0000 7fabf1d7b700 1 -- 172.21.15.7:0/10896953 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fabd80357e0 msgr2=0x7fabd8037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:53.184+0000 7fabf1d7b700 1 --2- 172.21.15.7:0/10896953 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fabd80357e0 0x7fabd8037c90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fabdc006fd0 tx=0x7fabdc007f90 comp rx=0 tx=0).stop 2023-09-30T01:57:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:53.184+0000 7fabf1d7b700 10 monclient: shutdown 2023-09-30T01:57:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:53.184+0000 7fabf1d7b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:53.184+0000 7fabf1d7b700 1 -- 172.21.15.7:0/10896953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabec137710 msgr2=0x7fabec09d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:53.184+0000 7fabf1d7b700 1 --2- 172.21.15.7:0/10896953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabec137710 0x7fabec09d100 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fabe4009ec0 tx=0x7fabe400b040 comp rx=0 tx=0).stop 2023-09-30T01:57:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:53.184+0000 7fabf1d7b700 1 -- 172.21.15.7:0/10896953 shutdown_connections 2023-09-30T01:57:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:53.184+0000 7fabf1d7b700 1 --2- 172.21.15.7:0/10896953 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fabd80357e0 0x7fabd8037c90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.185 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:53.184+0000 7fabf1d7b700 1 --2- 172.21.15.7:0/10896953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fabec137710 0x7fabec09d100 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:53.184+0000 7fabf1d7b700 1 --2- 172.21.15.7:0/10896953 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fabec096a50 0x7fabec0a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:53.184+0000 7fabf1d7b700 1 --2- 172.21.15.7:0/10896953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fabec096e20 0x7fabec09cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:53.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:53.184+0000 7fabf1d7b700 1 -- 172.21.15.7:0/10896953 >> 172.21.15.7:0/10896953 conn(0x7fabec070cf0 msgr2=0x7fabec12f110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:53.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:53.185+0000 7fabf1d7b700 1 -- 172.21.15.7:0/10896953 shutdown_connections 2023-09-30T01:57:53.186 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:53.185+0000 7fabf1d7b700 1 -- 172.21.15.7:0/10896953 wait complete. 2023-09-30T01:57:56.197 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:57:56.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.328+0000 7f931359e700 10 monclient: get_monmap_and_config 2023-09-30T01:57:56.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.328+0000 7f931359e700 10 monclient: build_initial_monmap 2023-09-30T01:57:56.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.328+0000 7f931359e700 10 monclient: monmap: 2023-09-30T01:57:56.329 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:56.329 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:56.329 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:57:56.329949+0000 2023-09-30T01:57:56.329 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:57:56.329949+0000 2023-09-30T01:57:56.329 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:56.329 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:56.330 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:57:56.330 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:57:56.330 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:57:56.330 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:57:56.330 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:57:56.330 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:56.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.329+0000 7f931359e700 1 Processor -- start 2023-09-30T01:57:56.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.329+0000 7f931359e700 1 -- start start 2023-09-30T01:57:56.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.329+0000 7f931359e700 10 monclient: init 2023-09-30T01:57:56.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931359e700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:56.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931359e700 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-09-30T01:57:56.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931359e700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9314098540 0x7f93141329e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:56.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931359e700 10 monclient(hunting): picked mon.noname-b con 0x7f9314098540 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:56.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931359e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9314098910 0x7f9314098ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:56.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931359e700 10 monclient(hunting): picked mon.noname-a con 0x7f9314098910 addr v2:172.21.15.7:3300/0 2023-09-30T01:57:56.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931359e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f931412ebc0 addr v1:172.21.15.50:6789/0 2023-09-30T01:57:56.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931359e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9314069f10 con 0x7f931412ebc0 2023-09-30T01:57:56.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931359e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:56.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931359e700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93140a0f40 con 0x7f9314098910 2023-09-30T01:57:56.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931359e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:56.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931359e700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9314132f20 con 0x7f9314098540 2023-09-30T01:57:56.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931359e700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:56.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931359e700 10 monclient(hunting): authenticate will time out at 1365.395225s 2023-09-30T01:57:56.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931259c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9314098540 0x7f93141329e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:56.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f9311d9b700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9314098910 0x7f9314098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:56.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931259c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9314098540 0x7f93141329e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:58540/0 (socket says 172.21.15.7:58540) 2023-09-30T01:57:56.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931259c700 1 -- 172.21.15.7:0/3106606020 learned_addr learned my addr 172.21.15.7:0/3106606020 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:56.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931259c700 10 monclient(hunting): get_auth_request con 0x7f9314098540 auth_method 0 2023-09-30T01:57:56.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:56.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:56.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:56.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f9311d9b700 10 monclient(hunting): get_auth_request con 0x7f9314098910 auth_method 0 2023-09-30T01:57:56.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f9311d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:56.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f9311d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:56.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f9311d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:56.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:56.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:56.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:56.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f9311d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:56.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f9311d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:56.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f9311d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:56.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931259c700 10 monclient(hunting): handle_auth_done global_id 4577 payload 274 2023-09-30T01:57:56.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931259c700 1 -- 172.21.15.7:0/3106606020 >> v2:172.21.15.7:3300/0 conn(0x7f9314098910 msgr2=0x7f9314098ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:56.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931259c700 1 --2- 172.21.15.7:0/3106606020 >> v2:172.21.15.7:3300/0 conn(0x7f9314098910 0x7f9314098ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:56.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931259c700 1 -- 172.21.15.7:0/3106606020 >> v1:172.21.15.50:6789/0 conn(0x7f931412ebc0 legacy=0x7f931412f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:56.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931259c700 10 monclient: _finish_hunting 0 2023-09-30T01:57:56.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931259c700 1 monclient: found mon.noname-b 2023-09-30T01:57:56.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:56.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931259c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:57:56.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f931259c700 1 -- 172.21.15.7:0/3106606020 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9314096a50 con 0x7f9314098540 2023-09-30T01:57:56.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.330+0000 7f9311d9b700 1 --2- 172.21.15.7:0/3106606020 >> v2:172.21.15.7:3300/0 conn(0x7f9314098910 0x7f9314098ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:56.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.332+0000 7f931259c700 1 -- 172.21.15.7:0/3106606020 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9314098540 msgr2=0x7f93141329e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:56.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.332+0000 7f931259c700 1 --2- 172.21.15.7:0/3106606020 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9314098540 0x7f93141329e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9304009f20 tx=0x7f9304009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b9dbe6cb40c3715 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:56.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931159a700 1 -- 172.21.15.7:0/3106606020 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930400e070 con 0x7f9314098540 2023-09-30T01:57:56.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:56.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:57:56.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931159a700 10 monclient: dump: 2023-09-30T01:57:56.342 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:56.342 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:56.342 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:56.342 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:56.342 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:56.342 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:56.342 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:56.342 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:56.342 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:56.343 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:56.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931159a700 10 monclient: _finish_auth 0 2023-09-30T01:57:56.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:56.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931359e700 5 monclient: authenticate success, global_id 4577 2023-09-30T01:57:56.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931159a700 1 -- 172.21.15.7:0/3106606020 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930400bba0 con 0x7f9314098540 2023-09-30T01:57:56.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:57:56.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:56.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931359e700 10 monclient: get_monmap_and_config success 2023-09-30T01:57:56.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931359e700 4 set_mon_vals no callback set 2023-09-30T01:57:56.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931159a700 1 -- 172.21.15.7:0/3106606020 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93040092e0 con 0x7f9314098540 2023-09-30T01:57:56.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:56.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:57:56.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931159a700 10 monclient: dump: 2023-09-30T01:57:56.344 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:56.344 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:56.344 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:56.344 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:56.344 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:56.344 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:56.344 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:56.344 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:56.344 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:56.345 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:56.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931359e700 10 monclient: shutdown 2023-09-30T01:57:56.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931359e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:56.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931359e700 1 -- 172.21.15.7:0/3106606020 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9314098540 msgr2=0x7f93141329e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:56.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931359e700 1 --2- 172.21.15.7:0/3106606020 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9314098540 0x7f93141329e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9304009f20 tx=0x7f9304009840 comp rx=0 tx=0).stop 2023-09-30T01:57:56.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931359e700 1 -- 172.21.15.7:0/3106606020 shutdown_connections 2023-09-30T01:57:56.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931359e700 1 --2- 172.21.15.7:0/3106606020 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9314098540 0x7f93141329e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:56.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931359e700 1 --2- 172.21.15.7:0/3106606020 >> v2:172.21.15.7:3300/0 conn(0x7f9314098910 0x7f9314098ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:56.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931359e700 1 -- 172.21.15.7:0/3106606020 >> 172.21.15.7:0/3106606020 conn(0x7f9314071830 msgr2=0x7f93140a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:56.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931359e700 1 -- 172.21.15.7:0/3106606020 shutdown_connections 2023-09-30T01:57:56.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.333+0000 7f931359e700 1 -- 172.21.15.7:0/3106606020 wait complete. 2023-09-30T01:57:56.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.334+0000 7f931359e700 10 monclient: build_initial_monmap 2023-09-30T01:57:56.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.334+0000 7f931359e700 10 monclient: monmap: 2023-09-30T01:57:56.346 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:56.346 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:57:56.346 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:57:56.346 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:57:56.346 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:56.346 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:56.346 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:57:56.346 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:57:56.346 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:57:56.347 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:56.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.334+0000 7f931359e700 1 Processor -- start 2023-09-30T01:57:56.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.334+0000 7f931359e700 1 -- start start 2023-09-30T01:57:56.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.334+0000 7f931359e700 10 monclient: init 2023-09-30T01:57:56.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.334+0000 7f931359e700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:56.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.334+0000 7f931359e700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:57:56.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.334+0000 7f931359e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9314098540 0x7f931409eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:56.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.334+0000 7f931359e700 10 monclient(hunting): picked mon.noname-b con 0x7f9314098540 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:57:56.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.334+0000 7f931359e700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9314098910 0x7f9314141b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:56.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.334+0000 7f931359e700 10 monclient(hunting): picked mon.noname-c con 0x7f9314098910 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:56.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.334+0000 7f931359e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f931412ebc0 0x7f931412e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:56.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.334+0000 7f931359e700 10 monclient(hunting): picked mon.noname-a con 0x7f931412ebc0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:57:56.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.334+0000 7f931359e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:56.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.334+0000 7f931359e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9314142080 con 0x7f931412ebc0 2023-09-30T01:57:56.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.334+0000 7f931359e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:56.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.334+0000 7f931359e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93141327b0 con 0x7f9314098540 2023-09-30T01:57:56.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.334+0000 7f931359e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:56.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.334+0000 7f931359e700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93140a4f40 con 0x7f9314098910 2023-09-30T01:57:56.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.334+0000 7f931359e700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:56.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.334+0000 7f931359e700 10 monclient(hunting): authenticate will time out at 1365.399992s 2023-09-30T01:57:56.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9312d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f931412ebc0 0x7f931412e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:56.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9311d9b700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9314098910 0x7f9314141b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:56.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9312d9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f931412ebc0 0x7f931412e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47556/0 (socket says 172.21.15.7:47556) 2023-09-30T01:57:56.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9312d9d700 1 -- 172.21.15.7:0/255715099 learned_addr learned my addr 172.21.15.7:0/255715099 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:56.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9312d9d700 10 monclient(hunting): get_auth_request con 0x7f931412ebc0 auth_method 0 2023-09-30T01:57:56.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9312d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:56.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9312d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:56.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9312d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:56.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9311d9b700 10 monclient(hunting): get_auth_request con 0x7f9314098910 auth_method 0 2023-09-30T01:57:56.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9311d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:56.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9311d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:56.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9311d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:56.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9311d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:56.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9311d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:56.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9311d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:56.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9312d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:56.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9312d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:56.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9312d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:56.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f931259c700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9314098540 0x7f931409eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:56.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f931259c700 10 monclient(hunting): get_auth_request con 0x7f9314098540 auth_method 0 2023-09-30T01:57:56.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f931259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:56.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f931259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:56.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f931259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:56.350 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f931259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:56.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f931259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:56.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f931259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:56.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9311d9b700 10 monclient(hunting): handle_auth_done global_id 4580 payload 824 2023-09-30T01:57:56.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9311d9b700 1 -- 172.21.15.7:0/255715099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9314098540 msgr2=0x7f931409eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:56.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9311d9b700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9314098540 0x7f931409eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:56.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9311d9b700 1 -- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f931412ebc0 msgr2=0x7f931412e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:56.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9311d9b700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f931412ebc0 0x7f931412e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:56.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9311d9b700 10 monclient: _finish_hunting 0 2023-09-30T01:57:56.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9311d9b700 1 monclient: found mon.noname-c 2023-09-30T01:57:56.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9311d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:56.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9311d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:57:56.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9311d9b700 1 -- 172.21.15.7:0/255715099 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92fc009420 con 0x7f9314098910 2023-09-30T01:57:56.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.335+0000 7f9312d9d700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f931412ebc0 0x7f931412e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:56.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f9311d9b700 1 -- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9314098910 msgr2=0x7f9314141b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:56.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f9311d9b700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9314098910 0x7f9314141b40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f92fc00ec80 tx=0x7f92fc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:56.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f930a7fc700 1 -- 172.21.15.7:0/255715099 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc009700 con 0x7f9314098910 2023-09-30T01:57:56.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f930a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:56.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f930a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:57:56.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f930a7fc700 10 monclient: dump: 2023-09-30T01:57:56.358 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:56.358 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:56.358 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:56.358 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:56.358 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:56.358 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:56.358 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:56.358 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:56.358 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:56.359 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:56.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f930a7fc700 10 monclient: _finish_auth 0 2023-09-30T01:57:56.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f930a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:56.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f931359e700 5 monclient: authenticate success, global_id 4580 2023-09-30T01:57:56.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f930a7fc700 1 -- 172.21.15.7:0/255715099 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92fc00ca60 con 0x7f9314098910 2023-09-30T01:57:56.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f930a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:56.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f930a7fc700 1 -- 172.21.15.7:0/255715099 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc00ce00 con 0x7f9314098910 2023-09-30T01:57:56.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f930a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:56.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f930a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:57:56.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f930a7fc700 10 monclient: dump: 2023-09-30T01:57:56.359 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:56.360 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:56.360 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:56.360 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:56.360 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:56.360 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:56.360 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:56.360 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:56.360 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:56.360 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:56.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f9310d99700 4 set_mon_vals no callback set 2023-09-30T01:57:56.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f931359e700 10 monclient: _renew_subs 2023-09-30T01:57:56.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f931359e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:56.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f931359e700 1 -- 172.21.15.7:0/255715099 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9304009ae0 con 0x7f9314098910 2023-09-30T01:57:56.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f931359e700 10 monclient: _renew_subs 2023-09-30T01:57:56.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f931359e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:56.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.337+0000 7f931359e700 1 -- 172.21.15.7:0/255715099 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93140a05c0 con 0x7f9314098910 2023-09-30T01:57:56.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9311d9b700 1 -- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9314098910 msgr2=0x7f9314141b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-09-30T01:57:56.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9311d9b700 1 -- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9314098910 msgr2=0x7f9314141b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-30T01:57:56.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9311d9b700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9314098910 0x7f9314141b40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f92fc00ec80 tx=0x7f92fc0091a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:57:56.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9311d9b700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9314098910 0x7f9314141b40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f92fc00ec80 tx=0x7f92fc0091a0 comp rx=0 tx=0).stop 2023-09-30T01:57:56.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f930a7fc700 10 monclient: ms_handle_reset current mon [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:56.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f930a7fc700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:56.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f930a7fc700 1 -- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9314098910 msgr2=0x7f9314141b40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-09-30T01:57:56.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f930a7fc700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9314098910 0x7f9314141b40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:56.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f930a7fc700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:57:56.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f930a7fc700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9314098540 0x7f92f4005860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:56.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f930a7fc700 10 monclient(hunting): picked mon.b con 0x7f9314098540 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:57:56.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f930a7fc700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f931412ebc0 0x7f92f4008190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:56.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f930a7fc700 10 monclient(hunting): picked mon.a con 0x7f931412ebc0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:57:56.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f930a7fc700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92f4008b10 0x7f92f400af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:56.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f930a7fc700 10 monclient(hunting): picked mon.c con 0x7f92f4008b10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:56.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f930a7fc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:56.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f930a7fc700 1 -- 172.21.15.7:0/255715099 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f400ba40 con 0x7f931412ebc0 2023-09-30T01:57:56.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f930a7fc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:56.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f930a7fc700 1 -- 172.21.15.7:0/255715099 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f400bb80 con 0x7f9314098540 2023-09-30T01:57:56.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f930a7fc700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:56.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f930a7fc700 1 -- 172.21.15.7:0/255715099 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f400bcc0 con 0x7f92f4008b10 2023-09-30T01:57:56.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f930a7fc700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:56.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9312d9d700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92f4008b10 0x7f92f400af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:56.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9312d9d700 10 monclient(hunting): get_auth_request con 0x7f92f4008b10 auth_method 0 2023-09-30T01:57:56.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9312d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:56.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9312d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:56.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9311d9b700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f931412ebc0 0x7f92f4008190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:56.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9312d9d700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:57:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9311d9b700 10 monclient(hunting): get_auth_request con 0x7f931412ebc0 auth_method 0 2023-09-30T01:57:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9311d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9311d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9311d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:57:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f931259c700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9314098540 0x7f92f4005860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f931259c700 10 monclient(hunting): get_auth_request con 0x7f9314098540 auth_method 0 2023-09-30T01:57:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f931259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f931259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f931259c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T01:57:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9312d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9312d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:56.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9312d9d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:57:56.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9311d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:56.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9311d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:56.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9311d9b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:57:56.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f931259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:56.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f931259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:56.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f931259c700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T01:57:56.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9312d9d700 10 monclient(hunting): handle_auth_done global_id 4580 payload 293 2023-09-30T01:57:56.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9312d9d700 1 -- 172.21.15.7:0/255715099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9314098540 msgr2=0x7f92f4005860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:56.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9312d9d700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9314098540 0x7f92f4005860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:56.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9312d9d700 1 -- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f931412ebc0 msgr2=0x7f92f4008190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:56.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9312d9d700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f931412ebc0 0x7f92f4008190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:56.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9312d9d700 10 monclient: _finish_hunting 0 2023-09-30T01:57:56.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9312d9d700 1 monclient: found mon.c 2023-09-30T01:57:56.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9312d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:56.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9312d9d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:56.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9312d9d700 1 -- 172.21.15.7:0/255715099 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f92f400bfb0 con 0x7f92f4008b10 2023-09-30T01:57:56.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9312d9d700 10 monclient: _finish_auth 0 2023-09-30T01:57:56.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9312d9d700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:56.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.338+0000 7f9311d9b700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f931412ebc0 0x7f92f4008190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:56.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.339+0000 7f931359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:56.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.339+0000 7f931359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:56.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.339+0000 7f931359e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:56.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.339+0000 7f931359e700 1 -- 172.21.15.7:0/255715099 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f931411ea90 con 0x7f92f4008b10 2023-09-30T01:57:56.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.340+0000 7f9312d9d700 1 -- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92f4008b10 msgr2=0x7f92f400af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:56.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.340+0000 7f9312d9d700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92f4008b10 0x7f92f400af90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f930c007fd0 tx=0x7f930c007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:56.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.341+0000 7f930a7fc700 1 -- 172.21.15.7:0/255715099 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c002820 con 0x7f92f4008b10 2023-09-30T01:57:56.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.341+0000 7f930a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:56.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.341+0000 7f930a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:57:56.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.341+0000 7f930a7fc700 10 monclient: dump: 2023-09-30T01:57:56.367 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:56.367 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:56.367 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:56.367 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:56.368 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:56.368 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:56.368 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:56.368 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:56.368 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:56.368 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:56.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.341+0000 7f930a7fc700 1 -- 172.21.15.7:0/255715099 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930c00fc00 con 0x7f92f4008b10 2023-09-30T01:57:56.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.341+0000 7f930a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:56.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.341+0000 7f930bfff700 4 set_mon_vals no callback set 2023-09-30T01:57:56.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.341+0000 7f930a7fc700 1 -- 172.21.15.7:0/255715099 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f930c00fd70 con 0x7f92f4008b10 2023-09-30T01:57:56.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.341+0000 7f930a7fc700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f92f403e2e0 0x7f92f4040790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:56.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.341+0000 7f930a7fc700 1 -- 172.21.15.7:0/255715099 <== mon.2 v2:172.21.15.7:3301/0 4 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f930c046370 con 0x7f92f4008b10 2023-09-30T01:57:56.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.342+0000 7f931259c700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f92f403e2e0 0x7f92f4040790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:56.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.342+0000 7f931259c700 10 monclient: get_auth_request con 0x7f92f403e2e0 auth_method 0 2023-09-30T01:57:56.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.344+0000 7f931259c700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f92f403e2e0 0x7f92f4040790 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9304009ab0 tx=0x7f9304017000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:56.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.346+0000 7f930a7fc700 1 -- 172.21.15.7:0/255715099 <== mon.2 v2:172.21.15.7:3301/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f930c012350 con 0x7f92f4008b10 2023-09-30T01:57:56.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.346+0000 7f930a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:56.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.346+0000 7f930a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:57:56.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.584+0000 7f931359e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:57:56.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.584+0000 7f931359e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:56.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.584+0000 7f931359e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:57:56.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.584+0000 7f931359e700 1 -- 172.21.15.7:0/255715099 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f931411e7a0 con 0x7f92f4008b10 2023-09-30T01:57:56.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.584+0000 7f930a7fc700 1 -- 172.21.15.7:0/255715099 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f930c021080 con 0x7f92f4008b10 2023-09-30T01:57:56.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.584+0000 7f930a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:56.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.584+0000 7f930a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:57:56.585 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:57:56.585 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:57:56.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.586+0000 7f931359e700 1 -- 172.21.15.7:0/255715099 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f92f403e2e0 msgr2=0x7f92f4040790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:56.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.586+0000 7f931359e700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f92f403e2e0 0x7f92f4040790 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9304009ab0 tx=0x7f9304017000 comp rx=0 tx=0).stop 2023-09-30T01:57:56.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.586+0000 7f931359e700 10 monclient: shutdown 2023-09-30T01:57:56.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.586+0000 7f931359e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:56.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.586+0000 7f931359e700 1 -- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92f4008b10 msgr2=0x7f92f400af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:56.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.586+0000 7f931359e700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92f4008b10 0x7f92f400af90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f930c007fd0 tx=0x7f930c007f60 comp rx=0 tx=0).stop 2023-09-30T01:57:56.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.586+0000 7f931359e700 1 -- 172.21.15.7:0/255715099 shutdown_connections 2023-09-30T01:57:56.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.586+0000 7f931359e700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f92f403e2e0 0x7f92f4040790 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:56.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.586+0000 7f931359e700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f931412ebc0 0x7f92f4008190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:56.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.586+0000 7f931359e700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92f4008b10 0x7f92f400af90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:56.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.586+0000 7f931359e700 1 --2- 172.21.15.7:0/255715099 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9314098540 0x7f92f4005860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:56.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.586+0000 7f931359e700 1 -- 172.21.15.7:0/255715099 >> 172.21.15.7:0/255715099 conn(0x7f9314071830 msgr2=0x7f93141317d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:56.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.586+0000 7f931359e700 1 -- 172.21.15.7:0/255715099 shutdown_connections 2023-09-30T01:57:56.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:56.586+0000 7f931359e700 1 -- 172.21.15.7:0/255715099 wait complete. 2023-09-30T01:57:59.598 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:57:59.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.730+0000 7fa852788700 10 monclient: get_monmap_and_config 2023-09-30T01:57:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.730+0000 7fa852788700 10 monclient: build_initial_monmap 2023-09-30T01:57:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.730+0000 7fa852788700 10 monclient: monmap: 2023-09-30T01:57:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:57:59.730851+0000 2023-09-30T01:57:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:57:59.730851+0000 2023-09-30T01:57:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:57:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:57:59.730 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:57:59.731 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:57:59.731 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:57:59.731 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:59.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.731+0000 7fa852788700 1 Processor -- start 2023-09-30T01:57:59.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.731+0000 7fa852788700 1 -- start start 2023-09-30T01:57:59.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.731+0000 7fa852788700 10 monclient: init 2023-09-30T01:57:59.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.731+0000 7fa852788700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:59.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.731+0000 7fa852788700 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-09-30T01:57:59.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa852788700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fa84c096a50 0x7fa84c1327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:59.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa852788700 10 monclient(hunting): picked mon.noname-c con 0x7fa84c096a50 addr v2:172.21.15.50:3300/0 2023-09-30T01:57:59.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa852788700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa84c0a2390 0x7fa84c096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:59.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa852788700 10 monclient(hunting): picked mon.noname-a con 0x7fa84c0a2390 addr v2:172.21.15.7:3300/0 2023-09-30T01:57:59.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa852788700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa84c097360 0x7fa84c12ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:59.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa852788700 10 monclient(hunting): picked mon.noname-b con 0x7fa84c097360 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:59.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa852788700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:59.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa852788700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa84c0980e0 con 0x7fa84c0a2390 2023-09-30T01:57:59.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa852788700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:59.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa852788700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa84c0991a0 con 0x7fa84c096a50 2023-09-30T01:57:59.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa852788700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:59.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa852788700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c099480 con 0x7fa84c097360 2023-09-30T01:57:59.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa852788700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:59.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa852788700 10 monclient(hunting): authenticate will time out at 1368.796069s 2023-09-30T01:57:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa84c0a2390 0x7fa84c096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa850d25700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa84c097360 0x7fa84c12ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84b7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa84c0a2390 0x7fa84c096e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47576/0 (socket says 172.21.15.7:47576) 2023-09-30T01:57:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84b7fe700 1 -- 172.21.15.7:0/3200870611 learned_addr learned my addr 172.21.15.7:0/3200870611 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa850d25700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa84c097360 0x7fa84c12ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:58572/0 (socket says 172.21.15.7:58572) 2023-09-30T01:57:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84b7fe700 10 monclient(hunting): get_auth_request con 0x7fa84c0a2390 auth_method 0 2023-09-30T01:57:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa850d25700 10 monclient(hunting): get_auth_request con 0x7fa84c097360 auth_method 0 2023-09-30T01:57:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa850d25700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa850d25700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa850d25700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa850d25700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa850d25700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa850d25700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84bfff700 1 --2- 172.21.15.7:0/3200870611 >> v2:172.21.15.50:3300/0 conn(0x7fa84c096a50 0x7fa84c1327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84b7fe700 10 monclient(hunting): handle_auth_done global_id 4569 payload 274 2023-09-30T01:57:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84b7fe700 1 -- 172.21.15.7:0/3200870611 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa84c097360 msgr2=0x7fa84c12ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:59.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84b7fe700 1 --2- 172.21.15.7:0/3200870611 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa84c097360 0x7fa84c12ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:59.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84b7fe700 1 -- 172.21.15.7:0/3200870611 >> v2:172.21.15.50:3300/0 conn(0x7fa84c096a50 msgr2=0x7fa84c1327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:57:59.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84b7fe700 1 --2- 172.21.15.7:0/3200870611 >> v2:172.21.15.50:3300/0 conn(0x7fa84c096a50 0x7fa84c1327a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:59.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84b7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:57:59.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84b7fe700 1 monclient: found mon.noname-a 2023-09-30T01:57:59.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:59.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:57:59.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84b7fe700 1 -- 172.21.15.7:0/3200870611 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84c097bf0 con 0x7fa84c0a2390 2023-09-30T01:57:59.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84bfff700 10 monclient: get_auth_request con 0x7fa84c096a50 auth_method 0 2023-09-30T01:57:59.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa850d25700 1 --2- 172.21.15.7:0/3200870611 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa84c097360 0x7fa84c12ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:59.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.732+0000 7fa84bfff700 1 --2- 172.21.15.7:0/3200870611 >> v2:172.21.15.50:3300/0 conn(0x7fa84c096a50 0x7fa84c1327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:57:59.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.734+0000 7fa84b7fe700 1 -- 172.21.15.7:0/3200870611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa84c0a2390 msgr2=0x7fa84c096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:59.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.734+0000 7fa84b7fe700 1 --2- 172.21.15.7:0/3200870611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa84c0a2390 0x7fa84c096e20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa84c0a2b00 tx=0x7fa83c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3363b2aedf41eb99 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:59.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.734+0000 7fa84affd700 1 -- 172.21.15.7:0/3200870611 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83c009e70 con 0x7fa84c0a2390 2023-09-30T01:57:59.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.734+0000 7fa84affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:59.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.734+0000 7fa84affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:57:59.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.734+0000 7fa84affd700 10 monclient: dump: 2023-09-30T01:57:59.742 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:59.742 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:59.742 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:59.742 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:59.742 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:59.743 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:59.743 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:59.743 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:59.743 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:59.743 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:59.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.734+0000 7fa84affd700 10 monclient: _finish_auth 0 2023-09-30T01:57:59.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.734+0000 7fa84affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:59.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.734+0000 7fa852788700 5 monclient: authenticate success, global_id 4569 2023-09-30T01:57:59.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.734+0000 7fa84affd700 1 -- 172.21.15.7:0/3200870611 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa83c00ba70 con 0x7fa84c0a2390 2023-09-30T01:57:59.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.734+0000 7fa84affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:59.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.734+0000 7fa852788700 10 monclient: get_monmap_and_config success 2023-09-30T01:57:59.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.734+0000 7fa852788700 4 set_mon_vals no callback set 2023-09-30T01:57:59.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.734+0000 7fa84affd700 1 -- 172.21.15.7:0/3200870611 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83c00be30 con 0x7fa84c0a2390 2023-09-30T01:57:59.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.734+0000 7fa84affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:59.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.734+0000 7fa84affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:57:59.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.734+0000 7fa84affd700 10 monclient: dump: 2023-09-30T01:57:59.744 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:59.744 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:59.744 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:59.744 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:59.744 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:59.744 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:59.745 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:59.745 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:59.745 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:59.745 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:59.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.735+0000 7fa852788700 10 monclient: shutdown 2023-09-30T01:57:59.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.735+0000 7fa852788700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:59.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.735+0000 7fa852788700 1 -- 172.21.15.7:0/3200870611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa84c0a2390 msgr2=0x7fa84c096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:59.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.735+0000 7fa852788700 1 --2- 172.21.15.7:0/3200870611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa84c0a2390 0x7fa84c096e20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa84c0a2b00 tx=0x7fa83c009250 comp rx=0 tx=0).stop 2023-09-30T01:57:59.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.735+0000 7fa852788700 1 -- 172.21.15.7:0/3200870611 shutdown_connections 2023-09-30T01:57:59.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.735+0000 7fa852788700 1 --2- 172.21.15.7:0/3200870611 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa84c097360 0x7fa84c12ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:59.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.735+0000 7fa852788700 1 --2- 172.21.15.7:0/3200870611 >> v2:172.21.15.50:3300/0 conn(0x7fa84c096a50 0x7fa84c1327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:59.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.735+0000 7fa852788700 1 --2- 172.21.15.7:0/3200870611 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa84c0a2390 0x7fa84c096e20 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.735+0000 7fa852788700 1 -- 172.21.15.7:0/3200870611 >> 172.21.15.7:0/3200870611 conn(0x7fa84c070cf0 msgr2=0x7fa84c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.735+0000 7fa852788700 1 -- 172.21.15.7:0/3200870611 shutdown_connections 2023-09-30T01:57:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.735+0000 7fa852788700 1 -- 172.21.15.7:0/3200870611 wait complete. 2023-09-30T01:57:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.736+0000 7fa852788700 10 monclient: build_initial_monmap 2023-09-30T01:57:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.736+0000 7fa852788700 10 monclient: monmap: 2023-09-30T01:57:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:57:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:57:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:57:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:57:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:57:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:59.746 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:57:59.747 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:57:59.747 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:57:59.747 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:59.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.736+0000 7fa852788700 1 Processor -- start 2023-09-30T01:57:59.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.736+0000 7fa852788700 1 -- start start 2023-09-30T01:57:59.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.736+0000 7fa852788700 10 monclient: init 2023-09-30T01:57:59.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.736+0000 7fa852788700 10 monclient: _reopen_session rank -1 2023-09-30T01:57:59.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.736+0000 7fa852788700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:57:59.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.736+0000 7fa852788700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa84c096a50 0x7fa84c141c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa852788700 10 monclient(hunting): picked mon.noname-a con 0x7fa84c096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:57:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa852788700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa84c097360 0x7fa84c142170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa852788700 10 monclient(hunting): picked mon.noname-b con 0x7fa84c097360 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:57:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa852788700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa84c0a2390 0x7fa84c09d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa852788700 10 monclient(hunting): picked mon.noname-c con 0x7fa84c0a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:57:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa852788700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa852788700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c133510 con 0x7fa84c096a50 2023-09-30T01:57:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa852788700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa852788700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c0a4c80 con 0x7fa84c097360 2023-09-30T01:57:59.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa852788700 10 monclient(hunting): start opening mon connection 2023-09-30T01:57:59.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa852788700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c0a15a0 con 0x7fa84c0a2390 2023-09-30T01:57:59.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa852788700 10 monclient(hunting): _renew_subs 2023-09-30T01:57:59.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa852788700 10 monclient(hunting): authenticate will time out at 1368.801070s 2023-09-30T01:57:59.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa84c096a50 0x7fa84c141c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:59.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa84c096a50 0x7fa84c141c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47584/0 (socket says 172.21.15.7:47584) 2023-09-30T01:57:59.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa850d25700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa84c0a2390 0x7fa84c09d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:59.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84bfff700 1 -- 172.21.15.7:0/3856098808 learned_addr learned my addr 172.21.15.7:0/3856098808 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:57:59.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84bfff700 10 monclient(hunting): get_auth_request con 0x7fa84c096a50 auth_method 0 2023-09-30T01:57:59.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:59.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:59.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:59.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa850d25700 10 monclient(hunting): get_auth_request con 0x7fa84c0a2390 auth_method 0 2023-09-30T01:57:59.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa850d25700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:57:59.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa850d25700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:57:59.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa850d25700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:57:59.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:59.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:59.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84b7fe700 1 --2- 172.21.15.7:0/3856098808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa84c097360 0x7fa84c142170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:59.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:59.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa850d25700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:57:59.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa850d25700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:57:59.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa850d25700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:57:59.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84bfff700 10 monclient(hunting): handle_auth_done global_id 4572 payload 824 2023-09-30T01:57:59.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84bfff700 1 -- 172.21.15.7:0/3856098808 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa84c0a2390 msgr2=0x7fa84c09d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:59.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84bfff700 1 --2- 172.21.15.7:0/3856098808 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa84c0a2390 0x7fa84c09d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:59.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84bfff700 1 -- 172.21.15.7:0/3856098808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa84c097360 msgr2=0x7fa84c142170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:59.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84bfff700 1 --2- 172.21.15.7:0/3856098808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa84c097360 0x7fa84c142170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:59.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84bfff700 10 monclient: _finish_hunting 0 2023-09-30T01:57:59.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84bfff700 1 monclient: found mon.noname-a 2023-09-30T01:57:59.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:57:59.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:57:59.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84bfff700 1 -- 172.21.15.7:0/3856098808 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa844009a20 con 0x7fa84c096a50 2023-09-30T01:57:59.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84b7fe700 10 monclient: get_auth_request con 0x7fa84c097360 auth_method 0 2023-09-30T01:57:59.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa850d25700 1 --2- 172.21.15.7:0/3856098808 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa84c0a2390 0x7fa84c09d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:57:59.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.737+0000 7fa84b7fe700 1 --2- 172.21.15.7:0/3856098808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa84c097360 0x7fa84c142170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:57:59.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.739+0000 7fa84bfff700 1 -- 172.21.15.7:0/3856098808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa84c096a50 msgr2=0x7fa84c141c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:57:59.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.739+0000 7fa84bfff700 1 --2- 172.21.15.7:0/3856098808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa84c096a50 0x7fa84c141c30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa84000c830 tx=0x7fa840008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:59.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.739+0000 7fa833fff700 1 -- 172.21.15.7:0/3856098808 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa840011810 con 0x7fa84c096a50 2023-09-30T01:57:59.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.739+0000 7fa833fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:59.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.739+0000 7fa833fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:57:59.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.739+0000 7fa833fff700 10 monclient: dump: 2023-09-30T01:57:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:59.759 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:59.760 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.739+0000 7fa833fff700 10 monclient: _finish_auth 0 2023-09-30T01:57:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.739+0000 7fa833fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:57:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.739+0000 7fa833fff700 1 -- 172.21.15.7:0/3856098808 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa840011970 con 0x7fa84c096a50 2023-09-30T01:57:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.739+0000 7fa852788700 5 monclient: authenticate success, global_id 4572 2023-09-30T01:57:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.739+0000 7fa852788700 10 monclient: _renew_subs 2023-09-30T01:57:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.739+0000 7fa852788700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.739+0000 7fa852788700 1 -- 172.21.15.7:0/3856098808 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa83c0094f0 con 0x7fa84c096a50 2023-09-30T01:57:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.739+0000 7fa833fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:57:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.739+0000 7fa833fff700 1 -- 172.21.15.7:0/3856098808 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa840011d30 con 0x7fa84c096a50 2023-09-30T01:57:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.740+0000 7fa833fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:57:59.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.740+0000 7fa833fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:57:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.740+0000 7fa833fff700 10 monclient: dump: 2023-09-30T01:57:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:57:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:57:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:57:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:57:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:57:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:57:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:57:59.761 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:57:59.761 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:57:59.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.740+0000 7fa84a7fc700 4 set_mon_vals no callback set 2023-09-30T01:57:59.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.740+0000 7fa852788700 10 monclient: _renew_subs 2023-09-30T01:57:59.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.740+0000 7fa852788700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:59.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.740+0000 7fa852788700 1 -- 172.21.15.7:0/3856098808 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84c09f4b0 con 0x7fa84c096a50 2023-09-30T01:57:59.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.740+0000 7fa833fff700 1 -- 172.21.15.7:0/3856098808 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fa84000a5f0 con 0x7fa84c096a50 2023-09-30T01:57:59.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.740+0000 7fa833fff700 1 --2- 172.21.15.7:0/3856098808 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa8380357e0 0x7fa838037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:57:59.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.741+0000 7fa833fff700 1 -- 172.21.15.7:0/3856098808 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fa840047df0 con 0x7fa84c096a50 2023-09-30T01:57:59.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.741+0000 7fa852788700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:59.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.741+0000 7fa852788700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:59.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.741+0000 7fa852788700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:59.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.741+0000 7fa852788700 1 -- 172.21.15.7:0/3856098808 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa84c064c10 con 0x7fa84c096a50 2023-09-30T01:57:59.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.741+0000 7fa84b7fe700 1 --2- 172.21.15.7:0/3856098808 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa8380357e0 0x7fa838037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:57:59.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.741+0000 7fa84b7fe700 10 monclient: get_auth_request con 0x7fa8380357e0 auth_method 0 2023-09-30T01:57:59.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.743+0000 7fa84b7fe700 1 --2- 172.21.15.7:0/3856098808 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa8380357e0 0x7fa838037c90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa84c0a2b00 tx=0x7fa83c017000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:57:59.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.745+0000 7fa833fff700 1 -- 172.21.15.7:0/3856098808 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa8400193b0 con 0x7fa84c096a50 2023-09-30T01:57:59.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.745+0000 7fa833fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:57:59.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.745+0000 7fa833fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:57:59.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.980+0000 7fa852788700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:57:59.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.980+0000 7fa852788700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:59.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.980+0000 7fa852788700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:57:59.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.980+0000 7fa852788700 1 -- 172.21.15.7:0/3856098808 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa84c132ce0 con 0x7fa84c096a50 2023-09-30T01:57:59.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.981+0000 7fa833fff700 1 -- 172.21.15.7:0/3856098808 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa840028080 con 0x7fa84c096a50 2023-09-30T01:57:59.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.981+0000 7fa833fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:57:59.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.981+0000 7fa833fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:57:59.981 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:57:59.981 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:57:59.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.983+0000 7fa852788700 1 -- 172.21.15.7:0/3856098808 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa8380357e0 msgr2=0x7fa838037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:59.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.983+0000 7fa852788700 1 --2- 172.21.15.7:0/3856098808 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa8380357e0 0x7fa838037c90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa84c0a2b00 tx=0x7fa83c017000 comp rx=0 tx=0).stop 2023-09-30T01:57:59.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.983+0000 7fa852788700 10 monclient: shutdown 2023-09-30T01:57:59.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.983+0000 7fa852788700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:57:59.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.983+0000 7fa852788700 1 -- 172.21.15.7:0/3856098808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa84c096a50 msgr2=0x7fa84c141c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:57:59.982 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.983+0000 7fa852788700 1 --2- 172.21.15.7:0/3856098808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa84c096a50 0x7fa84c141c30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa84000c830 tx=0x7fa840008040 comp rx=0 tx=0).stop 2023-09-30T01:57:59.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.983+0000 7fa852788700 1 -- 172.21.15.7:0/3856098808 shutdown_connections 2023-09-30T01:57:59.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.983+0000 7fa852788700 1 --2- 172.21.15.7:0/3856098808 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa8380357e0 0x7fa838037c90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:59.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.983+0000 7fa852788700 1 --2- 172.21.15.7:0/3856098808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa84c096a50 0x7fa84c141c30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:59.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.983+0000 7fa852788700 1 --2- 172.21.15.7:0/3856098808 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa84c0a2390 0x7fa84c09d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:59.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.983+0000 7fa852788700 1 --2- 172.21.15.7:0/3856098808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa84c097360 0x7fa84c142170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:57:59.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.983+0000 7fa852788700 1 -- 172.21.15.7:0/3856098808 >> 172.21.15.7:0/3856098808 conn(0x7fa84c070cf0 msgr2=0x7fa84c135700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:57:59.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.983+0000 7fa852788700 1 -- 172.21.15.7:0/3856098808 shutdown_connections 2023-09-30T01:57:59.983 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:57:59.983+0000 7fa852788700 1 -- 172.21.15.7:0/3856098808 wait complete. 2023-09-30T01:58:02.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:03.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.126+0000 7faa30ba1700 10 monclient: get_monmap_and_config 2023-09-30T01:58:03.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.126+0000 7faa30ba1700 10 monclient: build_initial_monmap 2023-09-30T01:58:03.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.126+0000 7faa30ba1700 10 monclient: monmap: 2023-09-30T01:58:03.126 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:03.126 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:03.126 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:58:03.127155+0000 2023-09-30T01:58:03.126 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:58:03.127155+0000 2023-09-30T01:58:03.127 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:03.127 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:03.127 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:58:03.127 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:58:03.127 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:58:03.127 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:58:03.127 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:58:03.127 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.126+0000 7faa30ba1700 1 Processor -- start 2023-09-30T01:58:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.126+0000 7faa30ba1700 1 -- start start 2023-09-30T01:58:03.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.126+0000 7faa30ba1700 10 monclient: init 2023-09-30T01:58:03.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa30ba1700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:03.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa30ba1700 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-09-30T01:58:03.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa30ba1700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faa2c096a50 0x7faa2c1327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:03.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa30ba1700 10 monclient(hunting): picked mon.noname-b con 0x7faa2c096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:03.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa30ba1700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7faa2c0a2390 0x7faa2c096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:03.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa30ba1700 10 monclient(hunting): picked mon.noname-c con 0x7faa2c0a2390 addr v2:172.21.15.50:3300/0 2023-09-30T01:58:03.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa30ba1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faa2c097360 0x7faa2c12ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:03.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa30ba1700 10 monclient(hunting): picked mon.noname-a con 0x7faa2c097360 addr v2:172.21.15.7:3300/0 2023-09-30T01:58:03.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa30ba1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:03.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa30ba1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa2c0980e0 con 0x7faa2c097360 2023-09-30T01:58:03.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa30ba1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa30ba1700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa2c0991a0 con 0x7faa2c0a2390 2023-09-30T01:58:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa30ba1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa30ba1700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c099480 con 0x7faa2c096a50 2023-09-30T01:58:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa30ba1700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa30ba1700 10 monclient(hunting): authenticate will time out at 1372.192414s 2023-09-30T01:58:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2a59c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faa2c096a50 0x7faa2c1327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2ad9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faa2c097360 0x7faa2c12ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2ad9d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7faa2c097360 0x7faa2c12ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47586/0 (socket says 172.21.15.7:47586) 2023-09-30T01:58:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2a59c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faa2c096a50 0x7faa2c1327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:58594/0 (socket says 172.21.15.7:58594) 2023-09-30T01:58:03.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2ad9d700 1 -- 172.21.15.7:0/121689859 learned_addr learned my addr 172.21.15.7:0/121689859 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:03.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2ad9d700 10 monclient(hunting): get_auth_request con 0x7faa2c097360 auth_method 0 2023-09-30T01:58:03.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:03.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:03.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:03.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2a59c700 10 monclient(hunting): get_auth_request con 0x7faa2c096a50 auth_method 0 2023-09-30T01:58:03.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:03.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:03.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:03.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa29d9b700 1 --2- 172.21.15.7:0/121689859 >> v2:172.21.15.50:3300/0 conn(0x7faa2c0a2390 0x7faa2c096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:03.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:03.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:03.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:03.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa29d9b700 10 monclient(hunting): get_auth_request con 0x7faa2c0a2390 auth_method 0 2023-09-30T01:58:03.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa29d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:03.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa29d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:03.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa29d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:03.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:03.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:03.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:03.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa29d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:03.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa29d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:03.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa29d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:03.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2a59c700 10 monclient(hunting): handle_auth_done global_id 4589 payload 274 2023-09-30T01:58:03.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2a59c700 1 -- 172.21.15.7:0/121689859 >> v2:172.21.15.50:3300/0 conn(0x7faa2c0a2390 msgr2=0x7faa2c096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:03.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2a59c700 1 --2- 172.21.15.7:0/121689859 >> v2:172.21.15.50:3300/0 conn(0x7faa2c0a2390 0x7faa2c096e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:03.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2a59c700 1 -- 172.21.15.7:0/121689859 >> v2:172.21.15.7:3300/0 conn(0x7faa2c097360 msgr2=0x7faa2c12ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:03.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2a59c700 1 --2- 172.21.15.7:0/121689859 >> v2:172.21.15.7:3300/0 conn(0x7faa2c097360 0x7faa2c12ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:03.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2a59c700 10 monclient: _finish_hunting 0 2023-09-30T01:58:03.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2a59c700 1 monclient: found mon.noname-b 2023-09-30T01:58:03.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:03.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2a59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:58:03.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2a59c700 1 -- 172.21.15.7:0/121689859 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa2c097bf0 con 0x7faa2c096a50 2023-09-30T01:58:03.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.127+0000 7faa2ad9d700 1 --2- 172.21.15.7:0/121689859 >> v2:172.21.15.7:3300/0 conn(0x7faa2c097360 0x7faa2c12ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:58:03.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa2a59c700 1 -- 172.21.15.7:0/121689859 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faa2c096a50 msgr2=0x7faa2c1327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:03.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa2a59c700 1 --2- 172.21.15.7:0/121689859 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faa2c096a50 0x7faa2c1327a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7faa2c06ad20 tx=0x7faa1c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7416d940bfc8cf1d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:03.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa2959a700 1 -- 172.21.15.7:0/121689859 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1c00e070 con 0x7faa2c096a50 2023-09-30T01:58:03.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa2959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:03.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa2959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:58:03.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa2959a700 10 monclient: dump: 2023-09-30T01:58:03.142 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:03.142 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:03.142 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:03.142 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:03.142 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:03.142 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:03.143 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:03.143 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:03.143 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:03.143 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:03.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa2959a700 10 monclient: _finish_auth 0 2023-09-30T01:58:03.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa2959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:03.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa30ba1700 5 monclient: authenticate success, global_id 4589 2023-09-30T01:58:03.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa30ba1700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:58:03.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa2959a700 1 -- 172.21.15.7:0/121689859 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1c00bcc0 con 0x7faa2c096a50 2023-09-30T01:58:03.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa2959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:03.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa30ba1700 10 monclient: get_monmap_and_config success 2023-09-30T01:58:03.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa30ba1700 4 set_mon_vals no callback set 2023-09-30T01:58:03.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa2959a700 1 -- 172.21.15.7:0/121689859 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1c012660 con 0x7faa2c096a50 2023-09-30T01:58:03.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa2959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:03.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa2959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:58:03.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa2959a700 10 monclient: dump: 2023-09-30T01:58:03.144 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:03.144 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:03.144 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:03.144 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:03.144 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:03.144 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:03.144 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:03.144 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:03.145 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:03.145 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:03.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa30ba1700 10 monclient: shutdown 2023-09-30T01:58:03.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa30ba1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:03.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa30ba1700 1 -- 172.21.15.7:0/121689859 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faa2c096a50 msgr2=0x7faa2c1327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:03.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.130+0000 7faa30ba1700 1 --2- 172.21.15.7:0/121689859 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faa2c096a50 0x7faa2c1327a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7faa2c06ad20 tx=0x7faa1c009780 comp rx=0 tx=0).stop 2023-09-30T01:58:03.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.131+0000 7faa30ba1700 1 -- 172.21.15.7:0/121689859 shutdown_connections 2023-09-30T01:58:03.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.131+0000 7faa30ba1700 1 --2- 172.21.15.7:0/121689859 >> v2:172.21.15.7:3300/0 conn(0x7faa2c097360 0x7faa2c12ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:03.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.131+0000 7faa30ba1700 1 --2- 172.21.15.7:0/121689859 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faa2c096a50 0x7faa2c1327a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:03.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.131+0000 7faa30ba1700 1 --2- 172.21.15.7:0/121689859 >> v2:172.21.15.50:3300/0 conn(0x7faa2c0a2390 0x7faa2c096e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:03.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.131+0000 7faa30ba1700 1 -- 172.21.15.7:0/121689859 >> 172.21.15.7:0/121689859 conn(0x7faa2c070cf0 msgr2=0x7faa2c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:03.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.131+0000 7faa30ba1700 1 -- 172.21.15.7:0/121689859 shutdown_connections 2023-09-30T01:58:03.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.131+0000 7faa30ba1700 1 -- 172.21.15.7:0/121689859 wait complete. 2023-09-30T01:58:03.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.131+0000 7faa30ba1700 10 monclient: build_initial_monmap 2023-09-30T01:58:03.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.131+0000 7faa30ba1700 10 monclient: monmap: 2023-09-30T01:58:03.146 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:03.146 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:58:03.146 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:58:03.146 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:58:03.146 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:03.146 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:03.146 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:58:03.146 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:58:03.147 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:58:03.147 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:03.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.131+0000 7faa30ba1700 1 Processor -- start 2023-09-30T01:58:03.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.131+0000 7faa30ba1700 1 -- start start 2023-09-30T01:58:03.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.131+0000 7faa30ba1700 10 monclient: init 2023-09-30T01:58:03.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa30ba1700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:03.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa30ba1700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:58:03.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa30ba1700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faa2c096a50 0x7faa2c12def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:03.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa30ba1700 10 monclient(hunting): picked mon.noname-c con 0x7faa2c096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:03.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa30ba1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa2c097360 0x7faa2c09da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:03.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa30ba1700 10 monclient(hunting): picked mon.noname-a con 0x7faa2c097360 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:58:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa30ba1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa2c0a2390 0x7faa2c09df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa30ba1700 10 monclient(hunting): picked mon.noname-b con 0x7faa2c0a2390 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:58:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa30ba1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa30ba1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c1334e0 con 0x7faa2c097360 2023-09-30T01:58:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa30ba1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa30ba1700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c0a4d70 con 0x7faa2c0a2390 2023-09-30T01:58:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa30ba1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa30ba1700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c1427f0 con 0x7faa2c096a50 2023-09-30T01:58:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa30ba1700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:03.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa30ba1700 10 monclient(hunting): authenticate will time out at 1372.197394s 2023-09-30T01:58:03.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2a59c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faa2c096a50 0x7faa2c12def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:03.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa29d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa2c097360 0x7faa2c09da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:03.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2a59c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faa2c096a50 0x7faa2c12def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:58608/0 (socket says 172.21.15.7:58608) 2023-09-30T01:58:03.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2a59c700 1 -- 172.21.15.7:0/1302132133 learned_addr learned my addr 172.21.15.7:0/1302132133 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:03.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2a59c700 10 monclient(hunting): get_auth_request con 0x7faa2c096a50 auth_method 0 2023-09-30T01:58:03.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:03.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:03.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:03.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa29d9b700 10 monclient(hunting): get_auth_request con 0x7faa2c097360 auth_method 0 2023-09-30T01:58:03.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa29d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:03.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa29d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:03.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa29d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:03.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:03.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:03.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:03.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa29d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:03.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa29d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:03.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa29d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:03.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2ad9d700 1 --2- 172.21.15.7:0/1302132133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa2c0a2390 0x7faa2c09df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:03.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2ad9d700 10 monclient(hunting): get_auth_request con 0x7faa2c0a2390 auth_method 0 2023-09-30T01:58:03.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:03.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:03.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2a59c700 10 monclient(hunting): handle_auth_done global_id 4592 payload 824 2023-09-30T01:58:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2a59c700 1 -- 172.21.15.7:0/1302132133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa2c0a2390 msgr2=0x7faa2c09df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2a59c700 1 --2- 172.21.15.7:0/1302132133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa2c0a2390 0x7faa2c09df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2a59c700 1 -- 172.21.15.7:0/1302132133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa2c097360 msgr2=0x7faa2c09da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2a59c700 1 --2- 172.21.15.7:0/1302132133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa2c097360 0x7faa2c09da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2a59c700 10 monclient: _finish_hunting 0 2023-09-30T01:58:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2a59c700 1 monclient: found mon.noname-c 2023-09-30T01:58:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2a59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:58:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2a59c700 1 -- 172.21.15.7:0/1302132133 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa240094f0 con 0x7faa2c096a50 2023-09-30T01:58:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.132+0000 7faa2ad9d700 1 --2- 172.21.15.7:0/1302132133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa2c0a2390 0x7faa2c09df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:58:03.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa2a59c700 1 -- 172.21.15.7:0/1302132133 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faa2c096a50 msgr2=0x7faa2c12def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa2a59c700 1 --2- 172.21.15.7:0/1302132133 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faa2c096a50 0x7faa2c12def0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7faa1c00b610 tx=0x7faa1c019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa227fc700 1 -- 172.21.15.7:0/1302132133 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1c012e30 con 0x7faa2c096a50 2023-09-30T01:58:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa227fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:58:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa227fc700 10 monclient: dump: 2023-09-30T01:58:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:03.152 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:03.153 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa227fc700 10 monclient: _finish_auth 0 2023-09-30T01:58:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa227fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa30ba1700 5 monclient: authenticate success, global_id 4592 2023-09-30T01:58:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa227fc700 1 -- 172.21.15.7:0/1302132133 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1c007000 con 0x7faa2c096a50 2023-09-30T01:58:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa30ba1700 10 monclient: _renew_subs 2023-09-30T01:58:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa30ba1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:03.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa30ba1700 1 -- 172.21.15.7:0/1302132133 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa1c009a20 con 0x7faa2c096a50 2023-09-30T01:58:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa227fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa227fc700 1 -- 172.21.15.7:0/1302132133 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1c026640 con 0x7faa2c096a50 2023-09-30T01:58:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa227fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:58:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa28d99700 4 set_mon_vals no callback set 2023-09-30T01:58:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa227fc700 10 monclient: dump: 2023-09-30T01:58:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:03.154 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:03.155 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa30ba1700 10 monclient: _renew_subs 2023-09-30T01:58:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa30ba1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa30ba1700 1 -- 172.21.15.7:0/1302132133 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa2c0a11f0 con 0x7faa2c096a50 2023-09-30T01:58:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.135+0000 7faa2a59c700 1 --2- 172.21.15.7:0/1302132133 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faa2c096a50 0x7faa2c12def0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7faa1c00b610 tx=0x7faa1c019000 comp rx=0 tx=0).queue_received will delay after 1.69604e+09 on 0x7faa1c026a70 mgrmap(e 5) v1 2023-09-30T01:58:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.136+0000 7faa227fc700 1 -- 172.21.15.7:0/1302132133 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7faa1c026a70 con 0x7faa2c096a50 2023-09-30T01:58:03.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.136+0000 7faa227fc700 1 --2- 172.21.15.7:0/1302132133 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7faa0c0357e0 0x7faa0c037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:03.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.136+0000 7faa30ba1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:03.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.136+0000 7faa30ba1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:03.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.136+0000 7faa30ba1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:03.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.136+0000 7faa30ba1700 1 -- 172.21.15.7:0/1302132133 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa2c0711e0 con 0x7faa2c096a50 2023-09-30T01:58:03.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.136+0000 7faa29d9b700 1 --2- 172.21.15.7:0/1302132133 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7faa0c0357e0 0x7faa0c037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:03.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.136+0000 7faa29d9b700 10 monclient: get_auth_request con 0x7faa0c0357e0 auth_method 0 2023-09-30T01:58:03.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.139+0000 7faa29d9b700 1 --2- 172.21.15.7:0/1302132133 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7faa0c0357e0 0x7faa0c037c90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7faa14009db0 tx=0x7faa140091a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:03.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.141+0000 7faa227fc700 1 -- 172.21.15.7:0/1302132133 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7faa1c0495e0 con 0x7faa2c096a50 2023-09-30T01:58:03.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.963+0000 7faa227fc700 1 -- 172.21.15.7:0/1302132133 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faa1c019310 con 0x7faa2c096a50 2023-09-30T01:58:03.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.963+0000 7faa227fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:03.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:03.963+0000 7faa227fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:58:04.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:04.210+0000 7faa30ba1700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:58:04.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:04.210+0000 7faa30ba1700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:04.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:04.210+0000 7faa30ba1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:04.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:04.210+0000 7faa30ba1700 1 -- 172.21.15.7:0/1302132133 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7faa2c0a1520 con 0x7faa2c096a50 2023-09-30T01:58:04.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:04.210+0000 7faa227fc700 1 -- 172.21.15.7:0/1302132133 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7faa1c02eb50 con 0x7faa2c096a50 2023-09-30T01:58:04.210 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:04.210+0000 7faa227fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:04.211 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:04.210+0000 7faa227fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:58:04.211 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:58:04.211 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:04.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:04.212+0000 7faa30ba1700 1 -- 172.21.15.7:0/1302132133 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7faa0c0357e0 msgr2=0x7faa0c037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:04.212 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:04.212+0000 7faa30ba1700 1 --2- 172.21.15.7:0/1302132133 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7faa0c0357e0 0x7faa0c037c90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7faa14009db0 tx=0x7faa140091a0 comp rx=0 tx=0).stop 2023-09-30T01:58:04.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:04.212+0000 7faa30ba1700 10 monclient: shutdown 2023-09-30T01:58:04.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:04.212+0000 7faa30ba1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:04.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:04.212+0000 7faa30ba1700 1 -- 172.21.15.7:0/1302132133 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faa2c096a50 msgr2=0x7faa2c12def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:04.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:04.212+0000 7faa30ba1700 1 --2- 172.21.15.7:0/1302132133 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faa2c096a50 0x7faa2c12def0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7faa1c00b610 tx=0x7faa1c019000 comp rx=0 tx=0).stop 2023-09-30T01:58:04.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:04.212+0000 7faa30ba1700 1 -- 172.21.15.7:0/1302132133 shutdown_connections 2023-09-30T01:58:04.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:04.212+0000 7faa30ba1700 1 --2- 172.21.15.7:0/1302132133 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7faa0c0357e0 0x7faa0c037c90 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:04.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:04.212+0000 7faa30ba1700 1 --2- 172.21.15.7:0/1302132133 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7faa2c0a2390 0x7faa2c09df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:04.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:04.212+0000 7faa30ba1700 1 --2- 172.21.15.7:0/1302132133 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7faa2c096a50 0x7faa2c12def0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:04.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:04.212+0000 7faa30ba1700 1 --2- 172.21.15.7:0/1302132133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa2c097360 0x7faa2c09da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:04.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:04.212+0000 7faa30ba1700 1 -- 172.21.15.7:0/1302132133 >> 172.21.15.7:0/1302132133 conn(0x7faa2c070cf0 msgr2=0x7faa2c1318e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:04.213 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:04.212+0000 7faa30ba1700 1 -- 172.21.15.7:0/1302132133 shutdown_connections 2023-09-30T01:58:04.214 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:04.212+0000 7faa30ba1700 1 -- 172.21.15.7:0/1302132133 wait complete. 2023-09-30T01:58:07.225 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:07.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.354+0000 7f278231d700 10 monclient: get_monmap_and_config 2023-09-30T01:58:07.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.354+0000 7f278231d700 10 monclient: build_initial_monmap 2023-09-30T01:58:07.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.354+0000 7f278231d700 10 monclient: monmap: 2023-09-30T01:58:07.355 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:07.355 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:07.355 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:58:07.355842+0000 2023-09-30T01:58:07.355 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:58:07.355842+0000 2023-09-30T01:58:07.355 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:07.355 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:07.355 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:58:07.355 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:58:07.356 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:58:07.356 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:58:07.356 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:58:07.356 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.355+0000 7f278231d700 1 Processor -- start 2023-09-30T01:58:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.355+0000 7f278231d700 1 -- start start 2023-09-30T01:58:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.355+0000 7f278231d700 10 monclient: init 2023-09-30T01:58:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278231d700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278231d700 10 monclient: _add_conns ranks=[4,3,0,2,1] 2023-09-30T01:58:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278231d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f277c098540 0x7f277c1329e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278231d700 10 monclient(hunting): picked mon.noname-c con 0x7f277c098540 addr v2:172.21.15.50:3300/0 2023-09-30T01:58:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278231d700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f277c098910 0x7f277c098d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278231d700 10 monclient(hunting): picked mon.noname-a con 0x7f277c098910 addr v2:172.21.15.7:3300/0 2023-09-30T01:58:07.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278231d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f277c12ebc0 addr v1:172.21.15.7:6789/0 2023-09-30T01:58:07.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278231d700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f277c069f10 con 0x7f277c12ebc0 2023-09-30T01:58:07.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278231d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:07.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278231d700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f277c0a0f40 con 0x7f277c098910 2023-09-30T01:58:07.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278231d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:07.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278231d700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f277c132f20 con 0x7f277c098540 2023-09-30T01:58:07.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278231d700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:07.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278231d700 10 monclient(hunting): authenticate will time out at 1376.421112s 2023-09-30T01:58:07.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f2781b1c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f277c12ebc0 0x7f277c12efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50650/0 (socket says 172.21.15.7:50650) 2023-09-30T01:58:07.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f2780b1a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f277c098910 0x7f277c098d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:07.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f2781b1c700 1 -- 172.21.15.7:0/1224373645 learned_addr learned my addr 172.21.15.7:0/1224373645 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:58:07.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f2780b1a700 10 monclient(hunting): get_auth_request con 0x7f277c098910 auth_method 0 2023-09-30T01:58:07.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f2780b1a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:07.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f2780b1a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:07.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f2780b1a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:07.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278131b700 1 --2- 172.21.15.7:0/1224373645 >> v2:172.21.15.50:3300/0 conn(0x7f277c098540 0x7f277c1329e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:07.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278131b700 10 monclient(hunting): get_auth_request con 0x7f277c098540 auth_method 0 2023-09-30T01:58:07.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278131b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:07.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278131b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:07.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278131b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:07.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f2780b1a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:07.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f2780b1a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:07.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f2780b1a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:07.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278131b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:07.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278131b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:07.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f278131b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:07.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f2780b1a700 10 monclient(hunting): handle_auth_done global_id 4581 payload 274 2023-09-30T01:58:07.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f2780b1a700 1 -- 172.21.15.7:0/1224373645 >> v2:172.21.15.50:3300/0 conn(0x7f277c098540 msgr2=0x7f277c1329e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:07.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f2780b1a700 1 --2- 172.21.15.7:0/1224373645 >> v2:172.21.15.50:3300/0 conn(0x7f277c098540 0x7f277c1329e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:07.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f2780b1a700 1 -- 172.21.15.7:0/1224373645 >> v1:172.21.15.7:6789/0 conn(0x7f277c12ebc0 legacy=0x7f277c12efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:07.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f2780b1a700 10 monclient: _finish_hunting 0 2023-09-30T01:58:07.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f2780b1a700 1 monclient: found mon.noname-a 2023-09-30T01:58:07.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f2780b1a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:07.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f2780b1a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:58:07.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.356+0000 7f2780b1a700 1 -- 172.21.15.7:0/1224373645 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f277c096a50 con 0x7f277c098910 2023-09-30T01:58:07.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.358+0000 7f2780b1a700 1 -- 172.21.15.7:0/1224373645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f277c098910 msgr2=0x7f277c098d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:07.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.358+0000 7f2780b1a700 1 --2- 172.21.15.7:0/1224373645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f277c098910 0x7f277c098d10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f277c071cb0 tx=0x7f27700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7acd9826b7ec48 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:07.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.359+0000 7f277bfff700 1 -- 172.21.15.7:0/1224373645 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2770011840 con 0x7f277c098910 2023-09-30T01:58:07.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.359+0000 7f277bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:07.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.359+0000 7f277bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:58:07.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.359+0000 7f277bfff700 10 monclient: dump: 2023-09-30T01:58:07.363 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:07.363 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:07.363 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:07.363 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:07.363 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:07.363 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:07.364 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:07.364 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:07.364 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:07.364 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:07.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.359+0000 7f277bfff700 10 monclient: _finish_auth 0 2023-09-30T01:58:07.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.359+0000 7f277bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:07.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.359+0000 7f278231d700 5 monclient: authenticate success, global_id 4581 2023-09-30T01:58:07.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.359+0000 7f278231d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:58:07.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.361+0000 7f277bfff700 1 -- 172.21.15.7:0/1224373645 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27700119a0 con 0x7f277c098910 2023-09-30T01:58:07.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.361+0000 7f277bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:07.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.361+0000 7f277bfff700 1 -- 172.21.15.7:0/1224373645 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2770011d40 con 0x7f277c098910 2023-09-30T01:58:07.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.361+0000 7f278231d700 10 monclient: get_monmap_and_config success 2023-09-30T01:58:07.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.361+0000 7f277bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:07.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.361+0000 7f278231d700 4 set_mon_vals no callback set 2023-09-30T01:58:07.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.361+0000 7f277bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:58:07.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.361+0000 7f277bfff700 10 monclient: dump: 2023-09-30T01:58:07.370 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:07.370 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:07.370 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:07.370 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:07.370 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:07.371 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:07.371 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:07.371 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:07.371 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:07.371 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:07.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.361+0000 7f278231d700 10 monclient: shutdown 2023-09-30T01:58:07.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.361+0000 7f278231d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:07.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.361+0000 7f278231d700 1 -- 172.21.15.7:0/1224373645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f277c098910 msgr2=0x7f277c098d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:07.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.361+0000 7f278231d700 1 --2- 172.21.15.7:0/1224373645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f277c098910 0x7f277c098d10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f277c071cb0 tx=0x7f27700091a0 comp rx=0 tx=0).stop 2023-09-30T01:58:07.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.361+0000 7f278231d700 1 -- 172.21.15.7:0/1224373645 shutdown_connections 2023-09-30T01:58:07.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.361+0000 7f278231d700 1 --2- 172.21.15.7:0/1224373645 >> v2:172.21.15.50:3300/0 conn(0x7f277c098540 0x7f277c1329e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:07.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.361+0000 7f278231d700 1 --2- 172.21.15.7:0/1224373645 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f277c098910 0x7f277c098d10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:07.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.361+0000 7f278231d700 1 -- 172.21.15.7:0/1224373645 >> 172.21.15.7:0/1224373645 conn(0x7f277c071830 msgr2=0x7f277c0a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:07.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.361+0000 7f278231d700 1 -- 172.21.15.7:0/1224373645 shutdown_connections 2023-09-30T01:58:07.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.361+0000 7f278231d700 1 -- 172.21.15.7:0/1224373645 wait complete. 2023-09-30T01:58:07.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.362+0000 7f278231d700 10 monclient: build_initial_monmap 2023-09-30T01:58:07.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.362+0000 7f278231d700 10 monclient: monmap: 2023-09-30T01:58:07.372 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:07.372 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:58:07.372 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:58:07.372 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:58:07.372 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:07.372 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:07.372 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:58:07.372 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:58:07.373 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:58:07.373 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:07.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.362+0000 7f278231d700 1 Processor -- start 2023-09-30T01:58:07.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.362+0000 7f278231d700 1 -- start start 2023-09-30T01:58:07.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.362+0000 7f278231d700 10 monclient: init 2023-09-30T01:58:07.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278231d700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:07.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278231d700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:58:07.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278231d700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f277c098540 0x7f277c09b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:07.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278231d700 10 monclient(hunting): picked mon.noname-c con 0x7f277c098540 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:07.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278231d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f277c098910 0x7f277c11ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:07.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278231d700 10 monclient(hunting): picked mon.noname-b con 0x7f277c098910 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:58:07.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278231d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f277c12ebc0 0x7f277c11bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:07.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278231d700 10 monclient(hunting): picked mon.noname-a con 0x7f277c12ebc0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:58:07.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278231d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:07.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278231d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277c12e720 con 0x7f277c12ebc0 2023-09-30T01:58:07.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278231d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:07.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278231d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277c11d5d0 con 0x7f277c098910 2023-09-30T01:58:07.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278231d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:07.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278231d700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f277c0a4800 con 0x7f277c098540 2023-09-30T01:58:07.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278231d700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:07.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278231d700 10 monclient(hunting): authenticate will time out at 1376.428336s 2023-09-30T01:58:07.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2781b1c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f277c12ebc0 0x7f277c11bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:07.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278131b700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f277c098540 0x7f277c09b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:07.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2781b1c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f277c12ebc0 0x7f277c11bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58012/0 (socket says 172.21.15.7:58012) 2023-09-30T01:58:07.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278131b700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f277c098540 0x7f277c09b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59854/0 (socket says 172.21.15.7:59854) 2023-09-30T01:58:07.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2781b1c700 1 -- 172.21.15.7:0/441680898 learned_addr learned my addr 172.21.15.7:0/441680898 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:07.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2781b1c700 10 monclient(hunting): get_auth_request con 0x7f277c12ebc0 auth_method 0 2023-09-30T01:58:07.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2781b1c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:07.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2781b1c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:07.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2781b1c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:07.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278131b700 10 monclient(hunting): get_auth_request con 0x7f277c098540 auth_method 0 2023-09-30T01:58:07.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278131b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:07.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278131b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:07.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278131b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:07.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2781b1c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:07.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2781b1c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:07.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2780b1a700 1 --2- 172.21.15.7:0/441680898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f277c098910 0x7f277c11ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:07.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2781b1c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:07.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2780b1a700 10 monclient(hunting): get_auth_request con 0x7f277c098910 auth_method 0 2023-09-30T01:58:07.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2780b1a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:07.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2780b1a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:07.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2780b1a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:07.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278131b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:07.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278131b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:07.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f278131b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:07.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2780b1a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:07.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2780b1a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:07.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2780b1a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:07.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2781b1c700 10 monclient(hunting): handle_auth_done global_id 4584 payload 824 2023-09-30T01:58:07.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2781b1c700 1 -- 172.21.15.7:0/441680898 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f277c098540 msgr2=0x7f277c09b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:07.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2781b1c700 1 --2- 172.21.15.7:0/441680898 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f277c098540 0x7f277c09b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:07.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2781b1c700 1 -- 172.21.15.7:0/441680898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f277c098910 msgr2=0x7f277c11ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:07.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2781b1c700 1 --2- 172.21.15.7:0/441680898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f277c098910 0x7f277c11ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:07.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2781b1c700 10 monclient: _finish_hunting 0 2023-09-30T01:58:07.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2781b1c700 1 monclient: found mon.noname-a 2023-09-30T01:58:07.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2781b1c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:07.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2781b1c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:58:07.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.363+0000 7f2781b1c700 1 -- 172.21.15.7:0/441680898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2770009420 con 0x7f277c12ebc0 2023-09-30T01:58:07.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f2781b1c700 1 -- 172.21.15.7:0/441680898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f277c12ebc0 msgr2=0x7f277c11bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:07.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f2781b1c700 1 --2- 172.21.15.7:0/441680898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f277c12ebc0 0x7f277c11bf60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f276c00ccf0 tx=0x7f276c005400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:07.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f27797fa700 1 -- 172.21.15.7:0/441680898 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c00aa80 con 0x7f277c12ebc0 2023-09-30T01:58:07.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f27797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:07.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f27797fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:58:07.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f27797fa700 10 monclient: dump: 2023-09-30T01:58:07.378 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:07.378 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:07.378 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:07.378 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:07.378 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:07.378 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:07.379 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:07.379 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:07.379 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:07.379 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:07.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f27797fa700 10 monclient: _finish_auth 0 2023-09-30T01:58:07.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f27797fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:07.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f27797fa700 1 -- 172.21.15.7:0/441680898 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f276c00ac00 con 0x7f277c12ebc0 2023-09-30T01:58:07.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f278231d700 5 monclient: authenticate success, global_id 4584 2023-09-30T01:58:07.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f278231d700 10 monclient: _renew_subs 2023-09-30T01:58:07.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f278231d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:07.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f278231d700 1 -- 172.21.15.7:0/441680898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f277c0a05c0 con 0x7f277c12ebc0 2023-09-30T01:58:07.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f27797fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:07.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f27797fa700 1 -- 172.21.15.7:0/441680898 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c0070f0 con 0x7f277c12ebc0 2023-09-30T01:58:07.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f27797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:07.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f27797fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:58:07.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f27797fa700 10 monclient: dump: 2023-09-30T01:58:07.380 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:07.380 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:07.380 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:07.380 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:07.380 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:07.380 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:07.380 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:07.381 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:07.381 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:07.381 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:07.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f277b7fe700 4 set_mon_vals no callback set 2023-09-30T01:58:07.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f278231d700 10 monclient: _renew_subs 2023-09-30T01:58:07.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f278231d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:07.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.366+0000 7f278231d700 1 -- 172.21.15.7:0/441680898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f277c11e690 con 0x7f277c12ebc0 2023-09-30T01:58:07.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.367+0000 7f27797fa700 1 -- 172.21.15.7:0/441680898 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f276c007460 con 0x7f277c12ebc0 2023-09-30T01:58:07.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.367+0000 7f278231d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:07.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.367+0000 7f278231d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:07.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.367+0000 7f278231d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:07.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.367+0000 7f278231d700 1 -- 172.21.15.7:0/441680898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f277c06a0e0 con 0x7f277c12ebc0 2023-09-30T01:58:07.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.367+0000 7f27797fa700 1 --2- 172.21.15.7:0/441680898 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2768035830 0x7f2768037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:07.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.367+0000 7f27797fa700 1 -- 172.21.15.7:0/441680898 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f276c048090 con 0x7f277c12ebc0 2023-09-30T01:58:07.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.367+0000 7f278131b700 1 --2- 172.21.15.7:0/441680898 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2768035830 0x7f2768037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:07.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.367+0000 7f278131b700 10 monclient: get_auth_request con 0x7f2768035830 auth_method 0 2023-09-30T01:58:07.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.370+0000 7f278131b700 1 --2- 172.21.15.7:0/441680898 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2768035830 0x7f2768037ce0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2774009d10 tx=0x7f27740091a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:07.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.372+0000 7f27797fa700 1 -- 172.21.15.7:0/441680898 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f276c010b30 con 0x7f277c12ebc0 2023-09-30T01:58:07.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.372+0000 7f27797fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:07.382 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.372+0000 7f27797fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:58:07.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.609+0000 7f278231d700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:58:07.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.609+0000 7f278231d700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:07.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.609+0000 7f278231d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:07.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.609+0000 7f278231d700 1 -- 172.21.15.7:0/441680898 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f277c064c10 con 0x7f277c12ebc0 2023-09-30T01:58:07.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.609+0000 7f27797fa700 1 -- 172.21.15.7:0/441680898 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f276c010300 con 0x7f277c12ebc0 2023-09-30T01:58:07.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.609+0000 7f27797fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:07.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.609+0000 7f27797fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:58:07.610 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:58:07.611 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:07.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.611+0000 7f278231d700 1 -- 172.21.15.7:0/441680898 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2768035830 msgr2=0x7f2768037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:07.615 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.611+0000 7f278231d700 1 --2- 172.21.15.7:0/441680898 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2768035830 0x7f2768037ce0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2774009d10 tx=0x7f27740091a0 comp rx=0 tx=0).stop 2023-09-30T01:58:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.611+0000 7f278231d700 10 monclient: shutdown 2023-09-30T01:58:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.611+0000 7f278231d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.611+0000 7f278231d700 1 -- 172.21.15.7:0/441680898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f277c12ebc0 msgr2=0x7f277c11bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.611+0000 7f278231d700 1 --2- 172.21.15.7:0/441680898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f277c12ebc0 0x7f277c11bf60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f276c00ccf0 tx=0x7f276c005400 comp rx=0 tx=0).stop 2023-09-30T01:58:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.612+0000 7f278231d700 1 -- 172.21.15.7:0/441680898 shutdown_connections 2023-09-30T01:58:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.612+0000 7f278231d700 1 --2- 172.21.15.7:0/441680898 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2768035830 0x7f2768037ce0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.612+0000 7f278231d700 1 --2- 172.21.15.7:0/441680898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f277c12ebc0 0x7f277c11bf60 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.612+0000 7f278231d700 1 --2- 172.21.15.7:0/441680898 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f277c098540 0x7f277c09b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.612+0000 7f278231d700 1 --2- 172.21.15.7:0/441680898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f277c098910 0x7f277c11ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.612+0000 7f278231d700 1 -- 172.21.15.7:0/441680898 >> 172.21.15.7:0/441680898 conn(0x7f277c071830 msgr2=0x7f277c1316d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:07.616 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.612+0000 7f278231d700 1 -- 172.21.15.7:0/441680898 shutdown_connections 2023-09-30T01:58:07.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:07.612+0000 7f278231d700 1 -- 172.21.15.7:0/441680898 wait complete. 2023-09-30T01:58:10.624 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:10.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.754+0000 7f993f1b9700 10 monclient: get_monmap_and_config 2023-09-30T01:58:10.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.754+0000 7f993f1b9700 10 monclient: build_initial_monmap 2023-09-30T01:58:10.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.755+0000 7f993f1b9700 10 monclient: monmap: 2023-09-30T01:58:10.755 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:10.755 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:10.755 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:58:10.756040+0000 2023-09-30T01:58:10.755 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:58:10.756040+0000 2023-09-30T01:58:10.755 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:10.755 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:10.756 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:58:10.756 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:58:10.756 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:58:10.756 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:58:10.756 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:58:10.756 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:10.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.755+0000 7f993f1b9700 1 Processor -- start 2023-09-30T01:58:10.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.755+0000 7f993f1b9700 1 -- start start 2023-09-30T01:58:10.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.755+0000 7f993f1b9700 10 monclient: init 2023-09-30T01:58:10.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993f1b9700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:10.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993f1b9700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2023-09-30T01:58:10.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993f1b9700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9938099860 0x7f9938099c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:10.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993f1b9700 10 monclient(hunting): picked mon.noname-b con 0x7f9938099860 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:10.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993f1b9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f993809e320 addr v1:172.21.15.7:6789/0 2023-09-30T01:58:10.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993f1b9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f993809e6f0 addr v1:172.21.15.50:6789/0 2023-09-30T01:58:10.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993f1b9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f993809e070 con 0x7f993809e320 2023-09-30T01:58:10.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993f1b9700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9938097e20 con 0x7f993809e6f0 2023-09-30T01:58:10.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993f1b9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:10.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993f1b9700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9938097570 con 0x7f9938099860 2023-09-30T01:58:10.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993f1b9700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:10.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993f1b9700 10 monclient(hunting): authenticate will time out at 1379.821400s 2023-09-30T01:58:10.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993cf55700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9938099860 0x7f9938099c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:10.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f9937fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f993809e320 0x7f993812fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50658/0 (socket says 172.21.15.7:50658) 2023-09-30T01:58:10.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f9937fff700 1 -- 172.21.15.7:0/644843613 learned_addr learned my addr 172.21.15.7:0/644843613 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:58:10.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993cf55700 10 monclient(hunting): get_auth_request con 0x7f9938099860 auth_method 0 2023-09-30T01:58:10.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993cf55700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:10.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993cf55700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:10.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993cf55700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:10.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993cf55700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:10.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993cf55700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:10.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993cf55700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:10.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993cf55700 10 monclient(hunting): handle_auth_done global_id 4598 payload 274 2023-09-30T01:58:10.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993cf55700 1 -- 172.21.15.7:0/644843613 >> v1:172.21.15.50:6789/0 conn(0x7f993809e6f0 legacy=0x7f99381305b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:10.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993cf55700 1 -- 172.21.15.7:0/644843613 >> v1:172.21.15.7:6789/0 conn(0x7f993809e320 legacy=0x7f993812fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:10.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993cf55700 10 monclient: _finish_hunting 0 2023-09-30T01:58:10.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993cf55700 1 monclient: found mon.noname-b 2023-09-30T01:58:10.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993cf55700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:10.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993cf55700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:58:10.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.756+0000 7f993cf55700 1 -- 172.21.15.7:0/644843613 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9938130cc0 con 0x7f9938099860 2023-09-30T01:58:10.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f993cf55700 1 -- 172.21.15.7:0/644843613 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9938099860 msgr2=0x7f9938099c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:10.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f993cf55700 1 --2- 172.21.15.7:0/644843613 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9938099860 0x7f9938099c30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9920009910 tx=0x7f99200091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2718e5520a8beb1b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:10.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f99377fe700 1 -- 172.21.15.7:0/644843613 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992000b9f0 con 0x7f9938099860 2023-09-30T01:58:10.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f99377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:10.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f99377fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:58:10.768 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f99377fe700 10 monclient: dump: 2023-09-30T01:58:10.768 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:10.768 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:10.768 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:10.768 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:10.769 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:10.769 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:10.769 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:10.769 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:10.769 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:10.769 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:10.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f99377fe700 10 monclient: _finish_auth 0 2023-09-30T01:58:10.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f99377fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:10.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f99377fe700 1 -- 172.21.15.7:0/644843613 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f992000bb50 con 0x7f9938099860 2023-09-30T01:58:10.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f993f1b9700 5 monclient: authenticate success, global_id 4598 2023-09-30T01:58:10.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f993f1b9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:58:10.769 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f99377fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f99377fe700 1 -- 172.21.15.7:0/644843613 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9920011420 con 0x7f9938099860 2023-09-30T01:58:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f993f1b9700 10 monclient: get_monmap_and_config success 2023-09-30T01:58:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f993f1b9700 4 set_mon_vals no callback set 2023-09-30T01:58:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f99377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f99377fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:58:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f99377fe700 10 monclient: dump: 2023-09-30T01:58:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:10.771 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:10.771 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:10.771 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:10.771 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:10.771 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:10.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f993f1b9700 10 monclient: shutdown 2023-09-30T01:58:10.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f993f1b9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:10.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f993f1b9700 1 -- 172.21.15.7:0/644843613 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9938099860 msgr2=0x7f9938099c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:10.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f993f1b9700 1 --2- 172.21.15.7:0/644843613 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9938099860 0x7f9938099c30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9920009910 tx=0x7f99200091a0 comp rx=0 tx=0).stop 2023-09-30T01:58:10.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f993f1b9700 1 -- 172.21.15.7:0/644843613 shutdown_connections 2023-09-30T01:58:10.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f993f1b9700 1 --2- 172.21.15.7:0/644843613 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9938099860 0x7f9938099c30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:10.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f993f1b9700 1 -- 172.21.15.7:0/644843613 >> 172.21.15.7:0/644843613 conn(0x7f9938067fd0 msgr2=0x7f99380683a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:10.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f993f1b9700 1 -- 172.21.15.7:0/644843613 shutdown_connections 2023-09-30T01:58:10.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.759+0000 7f993f1b9700 1 -- 172.21.15.7:0/644843613 wait complete. 2023-09-30T01:58:10.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.760+0000 7f993f1b9700 10 monclient: build_initial_monmap 2023-09-30T01:58:10.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.760+0000 7f993f1b9700 10 monclient: monmap: 2023-09-30T01:58:10.772 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:10.772 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:58:10.772 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:58:10.772 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:58:10.772 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:10.772 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:10.772 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:58:10.772 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:58:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:58:10.773 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.760+0000 7f993f1b9700 1 Processor -- start 2023-09-30T01:58:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.760+0000 7f993f1b9700 1 -- start start 2023-09-30T01:58:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.760+0000 7f993f1b9700 10 monclient: init 2023-09-30T01:58:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993f1b9700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993f1b9700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:58:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993f1b9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9938099860 0x7f993812d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993f1b9700 10 monclient(hunting): picked mon.noname-b con 0x7f9938099860 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:58:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993f1b9700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f993809e320 0x7f9938119ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993f1b9700 10 monclient(hunting): picked mon.noname-c con 0x7f993809e320 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993f1b9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f993809e6f0 0x7f993811a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993f1b9700 10 monclient(hunting): picked mon.noname-a con 0x7f993809e6f0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:58:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993f1b9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993f1b9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993811d530 con 0x7f993809e6f0 2023-09-30T01:58:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993f1b9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993f1b9700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993811e0f0 con 0x7f9938099860 2023-09-30T01:58:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993f1b9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993f1b9700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9938099630 con 0x7f993809e320 2023-09-30T01:58:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993f1b9700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993f1b9700 10 monclient(hunting): authenticate will time out at 1379.826481s 2023-09-30T01:58:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f9937fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f993809e320 0x7f9938119ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993d756700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f993809e6f0 0x7f993811a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f9937fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f993809e320 0x7f9938119ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:59880/0 (socket says 172.21.15.7:59880) 2023-09-30T01:58:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f9937fff700 1 -- 172.21.15.7:0/419527041 learned_addr learned my addr 172.21.15.7:0/419527041 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f9937fff700 10 monclient(hunting): get_auth_request con 0x7f993809e320 auth_method 0 2023-09-30T01:58:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f9937fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f9937fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f9937fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993d756700 10 monclient(hunting): get_auth_request con 0x7f993809e6f0 auth_method 0 2023-09-30T01:58:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993d756700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993d756700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993d756700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f9937fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:10.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f9937fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:10.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f9937fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:10.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993d756700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:10.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993d756700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:10.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993d756700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:10.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993cf55700 1 --2- 172.21.15.7:0/419527041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9938099860 0x7f993812d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:10.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993cf55700 10 monclient(hunting): get_auth_request con 0x7f9938099860 auth_method 0 2023-09-30T01:58:10.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993cf55700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:10.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993cf55700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:10.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993cf55700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:10.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f9937fff700 10 monclient(hunting): handle_auth_done global_id 4601 payload 824 2023-09-30T01:58:10.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f9937fff700 1 -- 172.21.15.7:0/419527041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9938099860 msgr2=0x7f993812d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:10.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f9937fff700 1 --2- 172.21.15.7:0/419527041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9938099860 0x7f993812d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:10.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f9937fff700 1 -- 172.21.15.7:0/419527041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f993809e6f0 msgr2=0x7f993811a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:10.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f9937fff700 1 --2- 172.21.15.7:0/419527041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f993809e6f0 0x7f993811a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:10.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f9937fff700 10 monclient: _finish_hunting 0 2023-09-30T01:58:10.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f9937fff700 1 monclient: found mon.noname-c 2023-09-30T01:58:10.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f9937fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:10.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f9937fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:58:10.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f9937fff700 1 -- 172.21.15.7:0/419527041 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9920009420 con 0x7f993809e320 2023-09-30T01:58:10.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.761+0000 7f993d756700 1 --2- 172.21.15.7:0/419527041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f993809e6f0 0x7f993811a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:58:10.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f9937fff700 1 -- 172.21.15.7:0/419527041 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f993809e320 msgr2=0x7f9938119ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:10.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f9937fff700 1 --2- 172.21.15.7:0/419527041 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f993809e320 0x7f9938119ec0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f992800c5e0 tx=0x7f9928004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:10.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f9934ff9700 1 -- 172.21.15.7:0/419527041 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9928005d40 con 0x7f993809e320 2023-09-30T01:58:10.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f9934ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:10.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f9934ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:58:10.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f9934ff9700 10 monclient: dump: 2023-09-30T01:58:10.778 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:10.778 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:10.778 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:10.778 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:10.778 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:10.778 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:10.778 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:10.779 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:10.779 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:10.779 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:10.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f9934ff9700 10 monclient: _finish_auth 0 2023-09-30T01:58:10.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f9934ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:10.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f9934ff9700 1 -- 172.21.15.7:0/419527041 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99280070d0 con 0x7f993809e320 2023-09-30T01:58:10.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f993f1b9700 5 monclient: authenticate success, global_id 4601 2023-09-30T01:58:10.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f993f1b9700 10 monclient: _renew_subs 2023-09-30T01:58:10.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f993f1b9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:10.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f993f1b9700 1 -- 172.21.15.7:0/419527041 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f993812fea0 con 0x7f993809e320 2023-09-30T01:58:10.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f9934ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:10.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f9934ff9700 1 -- 172.21.15.7:0/419527041 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9928007490 con 0x7f993809e320 2023-09-30T01:58:10.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f9934ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:10.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f9934ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:58:10.780 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f9934ff9700 10 monclient: dump: 2023-09-30T01:58:10.780 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:10.780 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:10.780 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:10.780 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:10.780 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:10.780 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:10.780 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:10.780 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:10.780 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:10.781 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:10.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f9936ffd700 4 set_mon_vals no callback set 2023-09-30T01:58:10.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f993f1b9700 10 monclient: _renew_subs 2023-09-30T01:58:10.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f993f1b9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:10.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.764+0000 7f993f1b9700 1 -- 172.21.15.7:0/419527041 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f993811b920 con 0x7f993809e320 2023-09-30T01:58:10.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.765+0000 7f9934ff9700 1 -- 172.21.15.7:0/419527041 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f992801b3b0 con 0x7f993809e320 2023-09-30T01:58:10.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.765+0000 7f993f1b9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:10.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.765+0000 7f993f1b9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:10.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.765+0000 7f993f1b9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:10.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.765+0000 7f993f1b9700 1 -- 172.21.15.7:0/419527041 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9938064c10 con 0x7f993809e320 2023-09-30T01:58:10.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.765+0000 7f9934ff9700 1 --2- 172.21.15.7:0/419527041 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9918035830 0x7f9918037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:10.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.765+0000 7f9934ff9700 1 -- 172.21.15.7:0/419527041 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f9928047320 con 0x7f993809e320 2023-09-30T01:58:10.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.765+0000 7f993cf55700 1 --2- 172.21.15.7:0/419527041 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9918035830 0x7f9918037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:10.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.765+0000 7f993cf55700 10 monclient: get_auth_request con 0x7f9918035830 auth_method 0 2023-09-30T01:58:10.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.768+0000 7f993cf55700 1 --2- 172.21.15.7:0/419527041 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9918035830 0x7f9918037ce0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f99200030c0 tx=0x7f9920016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:10.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.770+0000 7f9934ff9700 1 -- 172.21.15.7:0/419527041 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f992801be60 con 0x7f993809e320 2023-09-30T01:58:10.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.770+0000 7f9934ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:10.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:10.770+0000 7f9934ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:58:11.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:11.005+0000 7f993f1b9700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:58:11.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:11.005+0000 7f993f1b9700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:11.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:11.005+0000 7f993f1b9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:11.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:11.005+0000 7f993f1b9700 1 -- 172.21.15.7:0/419527041 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f993804bb40 con 0x7f993809e320 2023-09-30T01:58:11.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:11.006+0000 7f9934ff9700 1 -- 172.21.15.7:0/419527041 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9928017100 con 0x7f993809e320 2023-09-30T01:58:11.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:11.006+0000 7f9934ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:11.006 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:11.006+0000 7f9934ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:58:11.007 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:58:11.007 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:11.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:11.008+0000 7f993f1b9700 1 -- 172.21.15.7:0/419527041 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9918035830 msgr2=0x7f9918037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:11.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:11.008+0000 7f993f1b9700 1 --2- 172.21.15.7:0/419527041 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9918035830 0x7f9918037ce0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f99200030c0 tx=0x7f9920016000 comp rx=0 tx=0).stop 2023-09-30T01:58:11.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:11.008+0000 7f993f1b9700 10 monclient: shutdown 2023-09-30T01:58:11.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:11.008+0000 7f993f1b9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:11.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:11.008+0000 7f993f1b9700 1 -- 172.21.15.7:0/419527041 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f993809e320 msgr2=0x7f9938119ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:11.008 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:11.008+0000 7f993f1b9700 1 --2- 172.21.15.7:0/419527041 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f993809e320 0x7f9938119ec0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f992800c5e0 tx=0x7f9928004a80 comp rx=0 tx=0).stop 2023-09-30T01:58:11.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:11.008+0000 7f993f1b9700 1 -- 172.21.15.7:0/419527041 shutdown_connections 2023-09-30T01:58:11.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:11.008+0000 7f993f1b9700 1 --2- 172.21.15.7:0/419527041 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9918035830 0x7f9918037ce0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:11.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:11.008+0000 7f993f1b9700 1 --2- 172.21.15.7:0/419527041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f993809e6f0 0x7f993811a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:11.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:11.008+0000 7f993f1b9700 1 --2- 172.21.15.7:0/419527041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9938099860 0x7f993812d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:11.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:11.008+0000 7f993f1b9700 1 --2- 172.21.15.7:0/419527041 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f993809e320 0x7f9938119ec0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:11.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:11.008+0000 7f993f1b9700 1 -- 172.21.15.7:0/419527041 >> 172.21.15.7:0/419527041 conn(0x7f9938067fd0 msgr2=0x7f993812f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:11.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:11.008+0000 7f993f1b9700 1 -- 172.21.15.7:0/419527041 shutdown_connections 2023-09-30T01:58:11.009 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:11.008+0000 7f993f1b9700 1 -- 172.21.15.7:0/419527041 wait complete. 2023-09-30T01:58:14.021 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:14.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.150+0000 7fcd49fb7700 10 monclient: get_monmap_and_config 2023-09-30T01:58:14.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.150+0000 7fcd49fb7700 10 monclient: build_initial_monmap 2023-09-30T01:58:14.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.150+0000 7fcd49fb7700 10 monclient: monmap: 2023-09-30T01:58:14.151 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:14.151 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:14.151 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:58:14.151691+0000 2023-09-30T01:58:14.151 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:58:14.151691+0000 2023-09-30T01:58:14.151 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:14.151 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:14.151 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:58:14.151 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:58:14.151 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:58:14.151 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:58:14.152 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:58:14.152 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:14.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.151+0000 7fcd49fb7700 1 Processor -- start 2023-09-30T01:58:14.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.151+0000 7fcd49fb7700 1 -- start start 2023-09-30T01:58:14.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.151+0000 7fcd49fb7700 10 monclient: init 2023-09-30T01:58:14.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.151+0000 7fcd49fb7700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:14.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.151+0000 7fcd49fb7700 10 monclient: _add_conns ranks=[0,4,1,2,3] 2023-09-30T01:58:14.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.151+0000 7fcd49fb7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcd44096a50 addr v1:172.21.15.7:6789/0 2023-09-30T01:58:14.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.151+0000 7fcd49fb7700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcd440a2390 0x7fcd44096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:14.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd49fb7700 10 monclient(hunting): picked mon.noname-c con 0x7fcd440a2390 addr v2:172.21.15.50:3300/0 2023-09-30T01:58:14.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd49fb7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcd44097360 addr v1:172.21.15.50:6789/0 2023-09-30T01:58:14.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd49fb7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd440980e0 con 0x7fcd44096a50 2023-09-30T01:58:14.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd49fb7700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd44132eb0 con 0x7fcd44097360 2023-09-30T01:58:14.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd49fb7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:14.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd49fb7700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd440991a0 con 0x7fcd440a2390 2023-09-30T01:58:14.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd49fb7700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:14.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd49fb7700 10 monclient(hunting): authenticate will time out at 1383.217105s 2023-09-30T01:58:14.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd42ffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcd440a2390 0x7fcd44096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:14.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd42ffd700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fcd440a2390 0x7fcd44096e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.7:50532/0 (socket says 172.21.15.7:50532) 2023-09-30T01:58:14.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd42ffd700 1 -- 172.21.15.7:0/2504703978 learned_addr learned my addr 172.21.15.7:0/2504703978 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:14.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd42ffd700 10 monclient(hunting): get_auth_request con 0x7fcd440a2390 auth_method 0 2023-09-30T01:58:14.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd437fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fcd44096a50 0x7fcd441327a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54100/0 (socket says 172.21.15.7:54100) 2023-09-30T01:58:14.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd42ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:14.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd42ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:14.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd42ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:14.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd43fff700 1 --1- 172.21.15.7:0/2504703978 >> v1:172.21.15.50:6789/0 conn(0x7fcd44097360 0x7fcd44139720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.7:35770/0 (socket says 172.21.15.7:35770) 2023-09-30T01:58:14.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd42ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:14.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd42ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:14.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd42ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:14.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd42ffd700 10 monclient(hunting): handle_auth_done global_id 4525 payload 274 2023-09-30T01:58:14.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd42ffd700 1 -- 172.21.15.7:0/2504703978 >> v1:172.21.15.50:6789/0 conn(0x7fcd44097360 legacy=0x7fcd44139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:14.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd42ffd700 1 -- 172.21.15.7:0/2504703978 >> v1:172.21.15.7:6789/0 conn(0x7fcd44096a50 legacy=0x7fcd441327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:14.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd42ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:58:14.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd42ffd700 1 monclient: found mon.noname-c 2023-09-30T01:58:14.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd42ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:14.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd42ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3300/0 2023-09-30T01:58:14.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.152+0000 7fcd42ffd700 1 -- 172.21.15.7:0/2504703978 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd44099480 con 0x7fcd440a2390 2023-09-30T01:58:14.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.154+0000 7fcd42ffd700 1 -- 172.21.15.7:0/2504703978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd440a2390 msgr2=0x7fcd44096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:14.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.154+0000 7fcd42ffd700 1 --2- 172.21.15.7:0/2504703978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd440a2390 0x7fcd44096e20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcd380099b0 tx=0x7fcd38009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=757c356f8541417c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:14.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd427fc700 1 -- 172.21.15.7:0/2504703978 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3800bad0 con 0x7fcd440a2390 2023-09-30T01:58:14.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:14.163 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd427fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:58:14.164 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd427fc700 10 monclient: dump: 2023-09-30T01:58:14.164 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:14.164 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:14.164 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:14.164 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:14.164 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:14.164 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:14.164 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:14.164 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:14.165 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:14.165 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:14.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd427fc700 10 monclient: _finish_auth 0 2023-09-30T01:58:14.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd427fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:14.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd49fb7700 5 monclient: authenticate success, global_id 4525 2023-09-30T01:58:14.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd49fb7700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:58:14.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd427fc700 1 -- 172.21.15.7:0/2504703978 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd3800bc30 con 0x7fcd440a2390 2023-09-30T01:58:14.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd427fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:14.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd49fb7700 10 monclient: get_monmap_and_config success 2023-09-30T01:58:14.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd427fc700 1 -- 172.21.15.7:0/2504703978 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd38011420 con 0x7fcd440a2390 2023-09-30T01:58:14.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd49fb7700 4 set_mon_vals no callback set 2023-09-30T01:58:14.165 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:14.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd427fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:58:14.166 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd427fc700 10 monclient: dump: 2023-09-30T01:58:14.166 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:14.166 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:14.166 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:14.166 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:14.166 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:14.166 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:14.166 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:14.166 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:14.166 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:14.166 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd49fb7700 10 monclient: shutdown 2023-09-30T01:58:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd49fb7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd49fb7700 1 -- 172.21.15.7:0/2504703978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd440a2390 msgr2=0x7fcd44096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd49fb7700 1 --2- 172.21.15.7:0/2504703978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd440a2390 0x7fcd44096e20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcd380099b0 tx=0x7fcd38009250 comp rx=0 tx=0).stop 2023-09-30T01:58:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd49fb7700 1 -- 172.21.15.7:0/2504703978 shutdown_connections 2023-09-30T01:58:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd49fb7700 1 --2- 172.21.15.7:0/2504703978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd440a2390 0x7fcd44096e20 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd49fb7700 1 -- 172.21.15.7:0/2504703978 >> 172.21.15.7:0/2504703978 conn(0x7fcd44070cf0 msgr2=0x7fcd440710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd49fb7700 1 -- 172.21.15.7:0/2504703978 shutdown_connections 2023-09-30T01:58:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.155+0000 7fcd49fb7700 1 -- 172.21.15.7:0/2504703978 wait complete. 2023-09-30T01:58:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.156+0000 7fcd49fb7700 10 monclient: build_initial_monmap 2023-09-30T01:58:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.156+0000 7fcd49fb7700 10 monclient: monmap: 2023-09-30T01:58:14.167 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:14.168 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:58:14.168 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:58:14.168 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:58:14.168 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:14.168 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:14.168 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:58:14.168 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:58:14.169 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:58:14.169 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:14.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.156+0000 7fcd49fb7700 1 Processor -- start 2023-09-30T01:58:14.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.156+0000 7fcd49fb7700 1 -- start start 2023-09-30T01:58:14.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.156+0000 7fcd49fb7700 10 monclient: init 2023-09-30T01:58:14.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd49fb7700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:14.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd49fb7700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:58:14.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd49fb7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd44096a50 0x7fcd4409b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:14.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd49fb7700 10 monclient(hunting): picked mon.noname-a con 0x7fcd44096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:58:14.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd49fb7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd44097360 0x7fcd4409bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:14.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd49fb7700 10 monclient(hunting): picked mon.noname-b con 0x7fcd44097360 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:58:14.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd49fb7700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcd440a2390 0x7fcd4409c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:14.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd49fb7700 10 monclient(hunting): picked mon.noname-c con 0x7fcd440a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:14.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd49fb7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:14.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd49fb7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4409f130 con 0x7fcd44096a50 2023-09-30T01:58:14.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd49fb7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:14.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd49fb7700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd441334e0 con 0x7fcd44097360 2023-09-30T01:58:14.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd49fb7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:14.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd49fb7700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd441322c0 con 0x7fcd440a2390 2023-09-30T01:58:14.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd49fb7700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:14.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd49fb7700 10 monclient(hunting): authenticate will time out at 1383.222566s 2023-09-30T01:58:14.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd437fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd44096a50 0x7fcd4409b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:14.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd437fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd44096a50 0x7fcd4409b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47596/0 (socket says 172.21.15.7:47596) 2023-09-30T01:58:14.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd437fe700 1 -- 172.21.15.7:0/1779826288 learned_addr learned my addr 172.21.15.7:0/1779826288 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:14.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd437fe700 10 monclient(hunting): get_auth_request con 0x7fcd44096a50 auth_method 0 2023-09-30T01:58:14.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd437fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:14.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd437fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:14.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd437fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:14.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd43fff700 1 --2- 172.21.15.7:0/1779826288 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcd440a2390 0x7fcd4409c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:14.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd43fff700 10 monclient(hunting): get_auth_request con 0x7fcd440a2390 auth_method 0 2023-09-30T01:58:14.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd43fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:14.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd43fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:14.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd43fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:14.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd437fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:14.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd437fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:14.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd437fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:14.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd43fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:14.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd43fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:14.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd43fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:14.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.157+0000 7fcd43fff700 10 monclient(hunting): handle_auth_done global_id 4604 payload 824 2023-09-30T01:58:14.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.158+0000 7fcd43fff700 1 -- 172.21.15.7:0/1779826288 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd44097360 msgr2=0x7fcd4409bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:14.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.158+0000 7fcd43fff700 1 --2- 172.21.15.7:0/1779826288 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd44097360 0x7fcd4409bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:14.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.158+0000 7fcd43fff700 1 -- 172.21.15.7:0/1779826288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd44096a50 msgr2=0x7fcd4409b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:14.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.158+0000 7fcd43fff700 1 --2- 172.21.15.7:0/1779826288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd44096a50 0x7fcd4409b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:14.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.158+0000 7fcd43fff700 10 monclient: _finish_hunting 0 2023-09-30T01:58:14.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.158+0000 7fcd43fff700 1 monclient: found mon.noname-c 2023-09-30T01:58:14.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.158+0000 7fcd43fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:14.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.158+0000 7fcd43fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:58:14.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.158+0000 7fcd43fff700 1 -- 172.21.15.7:0/1779826288 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd380094f0 con 0x7fcd440a2390 2023-09-30T01:58:14.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.158+0000 7fcd437fe700 1 --2- 172.21.15.7:0/1779826288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd44096a50 0x7fcd4409b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:58:14.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd43fff700 1 -- 172.21.15.7:0/1779826288 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcd440a2390 msgr2=0x7fcd4409c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:14.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd43fff700 1 --2- 172.21.15.7:0/1779826288 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcd440a2390 0x7fcd4409c0c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c00ccc0 tx=0x7fcd3c005340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:14.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd2b7fe700 1 -- 172.21.15.7:0/1779826288 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c00a720 con 0x7fcd440a2390 2023-09-30T01:58:14.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd2b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:14.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd2b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:58:14.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd2b7fe700 10 monclient: dump: 2023-09-30T01:58:14.173 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:14.173 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:14.173 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:14.174 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:14.174 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:14.174 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:14.174 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:14.174 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:14.174 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:14.174 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:14.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd2b7fe700 10 monclient: _finish_auth 0 2023-09-30T01:58:14.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd2b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:14.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd49fb7700 5 monclient: authenticate success, global_id 4604 2023-09-30T01:58:14.174 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd49fb7700 10 monclient: _renew_subs 2023-09-30T01:58:14.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd2b7fe700 1 -- 172.21.15.7:0/1779826288 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd3c00a8a0 con 0x7fcd440a2390 2023-09-30T01:58:14.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd49fb7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:14.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd49fb7700 1 -- 172.21.15.7:0/1779826288 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd4409dfd0 con 0x7fcd440a2390 2023-09-30T01:58:14.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd2b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:14.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd2b7fe700 1 -- 172.21.15.7:0/1779826288 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c00ac60 con 0x7fcd440a2390 2023-09-30T01:58:14.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd2b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:14.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd2b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:58:14.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd2b7fe700 10 monclient: dump: 2023-09-30T01:58:14.175 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:14.175 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:14.175 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:14.175 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:14.176 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:14.176 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:14.176 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:14.176 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:14.176 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:14.176 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:14.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd49fb7700 10 monclient: _renew_subs 2023-09-30T01:58:14.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd41ffb700 4 set_mon_vals no callback set 2023-09-30T01:58:14.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd49fb7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:14.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.160+0000 7fcd49fb7700 1 -- 172.21.15.7:0/1779826288 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd440a1c10 con 0x7fcd440a2390 2023-09-30T01:58:14.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.161+0000 7fcd2b7fe700 1 -- 172.21.15.7:0/1779826288 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fcd3c007300 con 0x7fcd440a2390 2023-09-30T01:58:14.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.161+0000 7fcd2b7fe700 1 --2- 172.21.15.7:0/1779826288 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fcd300357e0 0x7fcd30037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:14.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.161+0000 7fcd49fb7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:14.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.161+0000 7fcd49fb7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:14.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.161+0000 7fcd49fb7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:14.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.161+0000 7fcd49fb7700 1 -- 172.21.15.7:0/1779826288 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd4409d220 con 0x7fcd440a2390 2023-09-30T01:58:14.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.161+0000 7fcd2b7fe700 1 -- 172.21.15.7:0/1779826288 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fcd3c047e80 con 0x7fcd440a2390 2023-09-30T01:58:14.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.161+0000 7fcd437fe700 1 --2- 172.21.15.7:0/1779826288 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fcd300357e0 0x7fcd30037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:14.177 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.161+0000 7fcd437fe700 10 monclient: get_auth_request con 0x7fcd300357e0 auth_method 0 2023-09-30T01:58:14.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.164+0000 7fcd437fe700 1 --2- 172.21.15.7:0/1779826288 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fcd300357e0 0x7fcd30037c90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcd34005140 tx=0x7fcd34007f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:14.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.166+0000 7fcd2b7fe700 1 -- 172.21.15.7:0/1779826288 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcd3c01b400 con 0x7fcd440a2390 2023-09-30T01:58:14.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.166+0000 7fcd2b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:14.178 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.166+0000 7fcd2b7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:58:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.400+0000 7fcd49fb7700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:58:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.400+0000 7fcd49fb7700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.400+0000 7fcd49fb7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.400+0000 7fcd49fb7700 1 -- 172.21.15.7:0/1779826288 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcd440a4cf0 con 0x7fcd440a2390 2023-09-30T01:58:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.401+0000 7fcd2b7fe700 1 -- 172.21.15.7:0/1779826288 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fcd3c010300 con 0x7fcd440a2390 2023-09-30T01:58:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.401+0000 7fcd2b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.401+0000 7fcd2b7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:58:14.401 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:58:14.402 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:14.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.403+0000 7fcd49fb7700 1 -- 172.21.15.7:0/1779826288 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fcd300357e0 msgr2=0x7fcd30037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:14.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.403+0000 7fcd49fb7700 1 --2- 172.21.15.7:0/1779826288 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fcd300357e0 0x7fcd30037c90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcd34005140 tx=0x7fcd34007f90 comp rx=0 tx=0).stop 2023-09-30T01:58:14.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.403+0000 7fcd49fb7700 10 monclient: shutdown 2023-09-30T01:58:14.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.403+0000 7fcd49fb7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:14.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.403+0000 7fcd49fb7700 1 -- 172.21.15.7:0/1779826288 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcd440a2390 msgr2=0x7fcd4409c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:14.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.403+0000 7fcd49fb7700 1 --2- 172.21.15.7:0/1779826288 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcd440a2390 0x7fcd4409c0c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c00ccc0 tx=0x7fcd3c005340 comp rx=0 tx=0).stop 2023-09-30T01:58:14.403 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.403+0000 7fcd49fb7700 1 -- 172.21.15.7:0/1779826288 shutdown_connections 2023-09-30T01:58:14.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.403+0000 7fcd49fb7700 1 --2- 172.21.15.7:0/1779826288 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fcd300357e0 0x7fcd30037c90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:14.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.403+0000 7fcd49fb7700 1 --2- 172.21.15.7:0/1779826288 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcd44096a50 0x7fcd4409b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:14.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.403+0000 7fcd49fb7700 1 --2- 172.21.15.7:0/1779826288 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fcd440a2390 0x7fcd4409c0c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:14.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.403+0000 7fcd49fb7700 1 --2- 172.21.15.7:0/1779826288 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcd44097360 0x7fcd4409bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:14.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.403+0000 7fcd49fb7700 1 -- 172.21.15.7:0/1779826288 >> 172.21.15.7:0/1779826288 conn(0x7fcd44070cf0 msgr2=0x7fcd440a3640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:14.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.403+0000 7fcd49fb7700 1 -- 172.21.15.7:0/1779826288 shutdown_connections 2023-09-30T01:58:14.404 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:14.403+0000 7fcd49fb7700 1 -- 172.21.15.7:0/1779826288 wait complete. 2023-09-30T01:58:17.416 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:17.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.545+0000 7fd56167f700 10 monclient: get_monmap_and_config 2023-09-30T01:58:17.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.545+0000 7fd56167f700 10 monclient: build_initial_monmap 2023-09-30T01:58:17.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.545+0000 7fd56167f700 10 monclient: monmap: 2023-09-30T01:58:17.546 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:17.546 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:17.546 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:58:17.546390+0000 2023-09-30T01:58:17.547 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:58:17.546390+0000 2023-09-30T01:58:17.547 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:17.547 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:17.547 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:58:17.547 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:58:17.548 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:58:17.548 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:58:17.548 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:58:17.548 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:17.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.545+0000 7fd56167f700 1 Processor -- start 2023-09-30T01:58:17.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.545+0000 7fd56167f700 1 -- start start 2023-09-30T01:58:17.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.545+0000 7fd56167f700 10 monclient: init 2023-09-30T01:58:17.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd56167f700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:17.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd56167f700 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-09-30T01:58:17.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd56167f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd55c0a29e0 addr v1:172.21.15.50:6789/0 2023-09-30T01:58:17.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd56167f700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd55c097160 0x7fd55c137550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:17.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd56167f700 10 monclient(hunting): picked mon.noname-b con 0x7fd55c097160 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:17.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd56167f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd55c097530 addr v1:172.21.15.7:6789/0 2023-09-30T01:58:17.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd56167f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55c069cc0 con 0x7fd55c097530 2023-09-30T01:58:17.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd56167f700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55c0a2db0 con 0x7fd55c0a29e0 2023-09-30T01:58:17.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd56167f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:17.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd56167f700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c098ea0 con 0x7fd55c097160 2023-09-30T01:58:17.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd56167f700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:17.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd56167f700 10 monclient(hunting): authenticate will time out at 1386.611536s 2023-09-30T01:58:17.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd55b7fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd55c097530 0x7fd55c139aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54112/0 (socket says 172.21.15.7:54112) 2023-09-30T01:58:17.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd55a7fc700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd55c097160 0x7fd55c137550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:17.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd55b7fe700 1 -- 172.21.15.7:0/3604608291 learned_addr learned my addr 172.21.15.7:0/3604608291 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:58:17.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd55a7fc700 10 monclient(hunting): get_auth_request con 0x7fd55c097160 auth_method 0 2023-09-30T01:58:17.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd55a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:17.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd55a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:17.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd55a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:17.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd55a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:17.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd55a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:17.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd55a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:17.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd55a7fc700 10 monclient(hunting): handle_auth_done global_id 4607 payload 274 2023-09-30T01:58:17.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd55a7fc700 1 -- 172.21.15.7:0/3604608291 >> v1:172.21.15.50:6789/0 conn(0x7fd55c0a29e0 legacy=0x7fd55c096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:17.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd55a7fc700 1 -- 172.21.15.7:0/3604608291 >> v1:172.21.15.7:6789/0 conn(0x7fd55c097530 legacy=0x7fd55c139aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:17.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd55a7fc700 10 monclient: _finish_hunting 0 2023-09-30T01:58:17.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd55a7fc700 1 monclient: found mon.noname-b 2023-09-30T01:58:17.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.546+0000 7fd55a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:17.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.547+0000 7fd55a7fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:58:17.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.547+0000 7fd55a7fc700 1 -- 172.21.15.7:0/3604608291 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55c098270 con 0x7fd55c097160 2023-09-30T01:58:17.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd55a7fc700 1 -- 172.21.15.7:0/3604608291 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd55c097160 msgr2=0x7fd55c137550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:17.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd55a7fc700 1 --2- 172.21.15.7:0/3604608291 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd55c097160 0x7fd55c137550 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd550009910 tx=0x7fd5500091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c6b4a1abdf0b29d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:17.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd559ffb700 1 -- 172.21.15.7:0/3604608291 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55000ba30 con 0x7fd55c097160 2023-09-30T01:58:17.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd559ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:17.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd559ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:58:17.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd559ffb700 10 monclient: dump: 2023-09-30T01:58:17.568 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:17.568 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:17.568 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:17.568 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:17.568 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:17.568 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:17.569 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:17.569 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:17.569 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:17.569 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:17.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd559ffb700 10 monclient: _finish_auth 0 2023-09-30T01:58:17.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd559ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:17.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd559ffb700 1 -- 172.21.15.7:0/3604608291 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55000bb90 con 0x7fd55c097160 2023-09-30T01:58:17.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd56167f700 5 monclient: authenticate success, global_id 4607 2023-09-30T01:58:17.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd56167f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:58:17.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd559ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:17.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd56167f700 10 monclient: get_monmap_and_config success 2023-09-30T01:58:17.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd56167f700 4 set_mon_vals no callback set 2023-09-30T01:58:17.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd559ffb700 1 -- 172.21.15.7:0/3604608291 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550011420 con 0x7fd55c097160 2023-09-30T01:58:17.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd559ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:17.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd559ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:58:17.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd559ffb700 10 monclient: dump: 2023-09-30T01:58:17.570 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:17.574 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:17.575 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:17.575 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:17.575 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:17.575 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:17.575 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:17.575 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:17.576 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:17.576 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:17.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd56167f700 10 monclient: shutdown 2023-09-30T01:58:17.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd56167f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:17.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd56167f700 1 -- 172.21.15.7:0/3604608291 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd55c097160 msgr2=0x7fd55c137550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:17.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd56167f700 1 --2- 172.21.15.7:0/3604608291 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd55c097160 0x7fd55c137550 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd550009910 tx=0x7fd5500091a0 comp rx=0 tx=0).stop 2023-09-30T01:58:17.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.549+0000 7fd56167f700 1 -- 172.21.15.7:0/3604608291 shutdown_connections 2023-09-30T01:58:17.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.550+0000 7fd56167f700 1 --2- 172.21.15.7:0/3604608291 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd55c097160 0x7fd55c137550 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:17.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.550+0000 7fd56167f700 1 -- 172.21.15.7:0/3604608291 >> 172.21.15.7:0/3604608291 conn(0x7fd55c0710d0 msgr2=0x7fd55c0714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:17.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.550+0000 7fd56167f700 1 -- 172.21.15.7:0/3604608291 shutdown_connections 2023-09-30T01:58:17.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.550+0000 7fd56167f700 1 -- 172.21.15.7:0/3604608291 wait complete. 2023-09-30T01:58:17.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.550+0000 7fd56167f700 10 monclient: build_initial_monmap 2023-09-30T01:58:17.577 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.550+0000 7fd56167f700 10 monclient: monmap: 2023-09-30T01:58:17.577 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:58:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:58:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:58:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:58:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:58:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:58:17.578 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd56167f700 1 Processor -- start 2023-09-30T01:58:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd56167f700 1 -- start start 2023-09-30T01:58:17.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd56167f700 10 monclient: init 2023-09-30T01:58:17.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd56167f700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:17.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd56167f700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:58:17.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd56167f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd55c097160 0x7fd55c141990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:17.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd56167f700 10 monclient(hunting): picked mon.noname-a con 0x7fd55c097160 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:58:17.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd56167f700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd55c097530 0x7fd55c12d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:17.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd56167f700 10 monclient(hunting): picked mon.noname-c con 0x7fd55c097530 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:17.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd56167f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd55c0a29e0 0x7fd55c09cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:17.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd56167f700 10 monclient(hunting): picked mon.noname-b con 0x7fd55c0a29e0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:58:17.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd56167f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:17.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd56167f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c142840 con 0x7fd55c097160 2023-09-30T01:58:17.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd56167f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:17.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd56167f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c09f870 con 0x7fd55c0a29e0 2023-09-30T01:58:17.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd56167f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:17.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd56167f700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c09f120 con 0x7fd55c097530 2023-09-30T01:58:17.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd56167f700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:17.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd56167f700 10 monclient(hunting): authenticate will time out at 1386.616677s 2023-09-30T01:58:17.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd55affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd55c097160 0x7fd55c141990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:17.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd55a7fc700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd55c097530 0x7fd55c12d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:17.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd55affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd55c097160 0x7fd55c141990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47610/0 (socket says 172.21.15.7:47610) 2023-09-30T01:58:17.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd55affd700 1 -- 172.21.15.7:0/3248103110 learned_addr learned my addr 172.21.15.7:0/3248103110 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:17.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd55affd700 10 monclient(hunting): get_auth_request con 0x7fd55c097160 auth_method 0 2023-09-30T01:58:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd55affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd55affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd55affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd55a7fc700 10 monclient(hunting): get_auth_request con 0x7fd55c097530 auth_method 0 2023-09-30T01:58:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd55a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd55a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd55a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd55affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:17.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd55affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd55affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd55a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd55a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd55a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.551+0000 7fd55b7fe700 1 --2- 172.21.15.7:0/3248103110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd55c0a29e0 0x7fd55c09cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.552+0000 7fd55affd700 10 monclient(hunting): handle_auth_done global_id 4593 payload 824 2023-09-30T01:58:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.552+0000 7fd55affd700 1 -- 172.21.15.7:0/3248103110 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd55c097530 msgr2=0x7fd55c12d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.552+0000 7fd55affd700 1 --2- 172.21.15.7:0/3248103110 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd55c097530 0x7fd55c12d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.552+0000 7fd55affd700 1 -- 172.21.15.7:0/3248103110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd55c0a29e0 msgr2=0x7fd55c09cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.552+0000 7fd55affd700 1 --2- 172.21.15.7:0/3248103110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd55c0a29e0 0x7fd55c09cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:17.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.552+0000 7fd55affd700 10 monclient: _finish_hunting 0 2023-09-30T01:58:17.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.552+0000 7fd55affd700 1 monclient: found mon.noname-a 2023-09-30T01:58:17.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.552+0000 7fd55affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:17.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.552+0000 7fd55affd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:58:17.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.552+0000 7fd55affd700 1 -- 172.21.15.7:0/3248103110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd550009420 con 0x7fd55c097160 2023-09-30T01:58:17.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.552+0000 7fd55b7fe700 10 monclient: get_auth_request con 0x7fd55c0a29e0 auth_method 0 2023-09-30T01:58:17.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.552+0000 7fd55b7fe700 1 --2- 172.21.15.7:0/3248103110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd55c0a29e0 0x7fd55c09cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:58:17.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.552+0000 7fd55a7fc700 1 --2- 172.21.15.7:0/3248103110 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd55c097530 0x7fd55c12d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:58:17.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd55affd700 1 -- 172.21.15.7:0/3248103110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd55c097160 msgr2=0x7fd55c141990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:17.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd55affd700 1 --2- 172.21.15.7:0/3248103110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd55c097160 0x7fd55c141990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd54c00c5b0 tx=0x7fd54c0020e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:17.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd542ffd700 1 -- 172.21.15.7:0/3248103110 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54c0117a0 con 0x7fd55c097160 2023-09-30T01:58:17.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd542ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:17.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd542ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:58:17.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd542ffd700 10 monclient: dump: 2023-09-30T01:58:17.584 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:17.584 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:17.584 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:17.584 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:17.584 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:17.584 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:17.584 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:17.584 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:17.584 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:17.584 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:17.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd542ffd700 10 monclient: _finish_auth 0 2023-09-30T01:58:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd542ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd542ffd700 1 -- 172.21.15.7:0/3248103110 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54c011920 con 0x7fd55c097160 2023-09-30T01:58:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd56167f700 5 monclient: authenticate success, global_id 4593 2023-09-30T01:58:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd56167f700 10 monclient: _renew_subs 2023-09-30T01:58:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd56167f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd56167f700 1 -- 172.21.15.7:0/3248103110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd55c0a06d0 con 0x7fd55c097160 2023-09-30T01:58:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd542ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd542ffd700 1 -- 172.21.15.7:0/3248103110 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54c011ce0 con 0x7fd55c097160 2023-09-30T01:58:17.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd542ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:17.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd542ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:58:17.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd542ffd700 10 monclient: dump: 2023-09-30T01:58:17.586 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:17.586 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:17.586 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:17.586 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:17.586 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:17.587 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:17.587 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:17.587 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:17.587 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:17.587 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:17.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd56167f700 10 monclient: _renew_subs 2023-09-30T01:58:17.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd5597fa700 4 set_mon_vals no callback set 2023-09-30T01:58:17.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd56167f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:17.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.554+0000 7fd56167f700 1 -- 172.21.15.7:0/3248103110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd55c09eba0 con 0x7fd55c097160 2023-09-30T01:58:17.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.555+0000 7fd56167f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:17.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.555+0000 7fd542ffd700 1 -- 172.21.15.7:0/3248103110 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fd54c012630 con 0x7fd55c097160 2023-09-30T01:58:17.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.555+0000 7fd56167f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:17.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.555+0000 7fd56167f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:17.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.555+0000 7fd56167f700 1 -- 172.21.15.7:0/3248103110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd55c064c10 con 0x7fd55c097160 2023-09-30T01:58:17.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.555+0000 7fd542ffd700 1 --2- 172.21.15.7:0/3248103110 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fd5480357e0 0x7fd548037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:17.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.555+0000 7fd542ffd700 1 -- 172.21.15.7:0/3248103110 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fd54c047a80 con 0x7fd55c097160 2023-09-30T01:58:17.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.556+0000 7fd55a7fc700 1 --2- 172.21.15.7:0/3248103110 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fd5480357e0 0x7fd548037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:17.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.556+0000 7fd55a7fc700 10 monclient: get_auth_request con 0x7fd5480357e0 auth_method 0 2023-09-30T01:58:17.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.558+0000 7fd55a7fc700 1 --2- 172.21.15.7:0/3248103110 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fd5480357e0 0x7fd548037c90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd5500093f0 tx=0x7fd550016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:17.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.560+0000 7fd542ffd700 1 -- 172.21.15.7:0/3248103110 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd54c002150 con 0x7fd55c097160 2023-09-30T01:58:17.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.560+0000 7fd542ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:17.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.560+0000 7fd542ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:58:17.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.796+0000 7fd56167f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:58:17.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.796+0000 7fd56167f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:17.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.796+0000 7fd56167f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:17.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.796+0000 7fd56167f700 1 -- 172.21.15.7:0/3248103110 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd55c04bb40 con 0x7fd55c097160 2023-09-30T01:58:17.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.796+0000 7fd542ffd700 1 -- 172.21.15.7:0/3248103110 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd54c0102e0 con 0x7fd55c097160 2023-09-30T01:58:17.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.796+0000 7fd542ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:17.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.796+0000 7fd542ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:58:17.797 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:58:17.797 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:17.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.798+0000 7fd56167f700 1 -- 172.21.15.7:0/3248103110 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fd5480357e0 msgr2=0x7fd548037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:17.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.798+0000 7fd56167f700 1 --2- 172.21.15.7:0/3248103110 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fd5480357e0 0x7fd548037c90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd5500093f0 tx=0x7fd550016000 comp rx=0 tx=0).stop 2023-09-30T01:58:17.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.798+0000 7fd56167f700 10 monclient: shutdown 2023-09-30T01:58:17.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.798+0000 7fd56167f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:17.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.798+0000 7fd56167f700 1 -- 172.21.15.7:0/3248103110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd55c097160 msgr2=0x7fd55c141990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:17.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.798+0000 7fd56167f700 1 --2- 172.21.15.7:0/3248103110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd55c097160 0x7fd55c141990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd54c00c5b0 tx=0x7fd54c0020e0 comp rx=0 tx=0).stop 2023-09-30T01:58:17.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.798+0000 7fd56167f700 1 -- 172.21.15.7:0/3248103110 shutdown_connections 2023-09-30T01:58:17.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.798+0000 7fd56167f700 1 --2- 172.21.15.7:0/3248103110 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fd5480357e0 0x7fd548037c90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:17.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.798+0000 7fd56167f700 1 --2- 172.21.15.7:0/3248103110 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd55c097160 0x7fd55c141990 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:17.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.798+0000 7fd56167f700 1 --2- 172.21.15.7:0/3248103110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd55c0a29e0 0x7fd55c09cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:17.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.798+0000 7fd56167f700 1 --2- 172.21.15.7:0/3248103110 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fd55c097530 0x7fd55c12d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:17.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.798+0000 7fd56167f700 1 -- 172.21.15.7:0/3248103110 >> 172.21.15.7:0/3248103110 conn(0x7fd55c0710d0 msgr2=0x7fd55c0a41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:17.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.798+0000 7fd56167f700 1 -- 172.21.15.7:0/3248103110 shutdown_connections 2023-09-30T01:58:17.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:17.798+0000 7fd56167f700 1 -- 172.21.15.7:0/3248103110 wait complete. 2023-09-30T01:58:20.812 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:20.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.943+0000 7f2c7bb46700 10 monclient: get_monmap_and_config 2023-09-30T01:58:20.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.943+0000 7f2c7bb46700 10 monclient: build_initial_monmap 2023-09-30T01:58:20.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.943+0000 7f2c7bb46700 10 monclient: monmap: 2023-09-30T01:58:20.943 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:20.943 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:20.944 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:58:20.944197+0000 2023-09-30T01:58:20.944 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:58:20.944197+0000 2023-09-30T01:58:20.944 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:20.944 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:20.944 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:58:20.944 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:58:20.944 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:58:20.944 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:58:20.944 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:58:20.944 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:20.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.943+0000 7f2c7bb46700 1 Processor -- start 2023-09-30T01:58:20.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.943+0000 7f2c7bb46700 1 -- start start 2023-09-30T01:58:20.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.943+0000 7f2c7bb46700 10 monclient: init 2023-09-30T01:58:20.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c7bb46700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:20.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c7bb46700 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-09-30T01:58:20.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c7bb46700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2c74096a50 0x7f2c74132680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:20.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c7bb46700 10 monclient(hunting): picked mon.noname-b con 0x7f2c74096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:20.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c7bb46700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2c74096e20 0x7f2c740971f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:20.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c7bb46700 10 monclient(hunting): picked mon.noname-a con 0x7f2c74096e20 addr v2:172.21.15.7:3300/0 2023-09-30T01:58:20.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c7bb46700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2c7412ef00 addr v1:172.21.15.50:6789/0 2023-09-30T01:58:20.952 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c7bb46700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c74099440 con 0x7f2c7412ef00 2023-09-30T01:58:20.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c7bb46700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:20.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c7bb46700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c74097b40 con 0x7f2c74096e20 2023-09-30T01:58:20.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c7bb46700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:20.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c7bb46700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c74099160 con 0x7f2c74096a50 2023-09-30T01:58:20.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c7bb46700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:20.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c7bb46700 10 monclient(hunting): authenticate will time out at 1390.009477s 2023-09-30T01:58:20.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c798e2700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2c74096a50 0x7f2c74132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:20.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c790e1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f2c74096e20 0x7f2c740971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:20.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c798e2700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2c74096a50 0x7f2c74132680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:55464/0 (socket says 172.21.15.7:55464) 2023-09-30T01:58:20.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c798e2700 1 -- 172.21.15.7:0/1639652431 learned_addr learned my addr 172.21.15.7:0/1639652431 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:20.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c798e2700 10 monclient(hunting): get_auth_request con 0x7f2c74096a50 auth_method 0 2023-09-30T01:58:20.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c798e2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:20.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c798e2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:20.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c798e2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:20.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c790e1700 10 monclient(hunting): get_auth_request con 0x7f2c74096e20 auth_method 0 2023-09-30T01:58:20.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c790e1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:20.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c790e1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:20.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c790e1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:20.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c798e2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:20.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c798e2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:20.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c798e2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:20.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c790e1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:20.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c790e1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:20.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c790e1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:20.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c798e2700 10 monclient(hunting): handle_auth_done global_id 4613 payload 274 2023-09-30T01:58:20.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c798e2700 1 -- 172.21.15.7:0/1639652431 >> v2:172.21.15.7:3300/0 conn(0x7f2c74096e20 msgr2=0x7f2c740971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:20.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c798e2700 1 --2- 172.21.15.7:0/1639652431 >> v2:172.21.15.7:3300/0 conn(0x7f2c74096e20 0x7f2c740971f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:20.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c798e2700 1 -- 172.21.15.7:0/1639652431 >> v1:172.21.15.50:6789/0 conn(0x7f2c7412ef00 legacy=0x7f2c7412f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:20.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c798e2700 10 monclient: _finish_hunting 0 2023-09-30T01:58:20.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c798e2700 1 monclient: found mon.noname-b 2023-09-30T01:58:20.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c798e2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:20.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c798e2700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:58:20.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.944+0000 7f2c798e2700 1 -- 172.21.15.7:0/1639652431 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c74099720 con 0x7f2c74096a50 2023-09-30T01:58:20.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c798e2700 1 -- 172.21.15.7:0/1639652431 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2c74096a50 msgr2=0x7f2c74132680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:20.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c798e2700 1 --2- 172.21.15.7:0/1639652431 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2c74096a50 0x7f2c74132680 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2c64009f20 tx=0x7f2c64009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f8c1cdbb1460224 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:20.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c788e0700 1 -- 172.21.15.7:0/1639652431 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6400e070 con 0x7f2c74096a50 2023-09-30T01:58:20.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c788e0700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:20.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c788e0700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:58:20.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c788e0700 10 monclient: dump: 2023-09-30T01:58:20.956 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:20.956 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:20.956 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:20.956 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:20.956 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:20.956 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:20.956 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:20.956 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:20.956 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:20.956 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:20.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c788e0700 10 monclient: _finish_auth 0 2023-09-30T01:58:20.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c788e0700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:20.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c7bb46700 5 monclient: authenticate success, global_id 4613 2023-09-30T01:58:20.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c788e0700 1 -- 172.21.15.7:0/1639652431 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c6400bba0 con 0x7f2c74096a50 2023-09-30T01:58:20.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c7bb46700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:58:20.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c788e0700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:20.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c7bb46700 10 monclient: get_monmap_and_config success 2023-09-30T01:58:20.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c788e0700 1 -- 172.21.15.7:0/1639652431 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c640092e0 con 0x7f2c74096a50 2023-09-30T01:58:20.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c7bb46700 4 set_mon_vals no callback set 2023-09-30T01:58:20.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c788e0700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:20.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c788e0700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:58:20.957 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c788e0700 10 monclient: dump: 2023-09-30T01:58:20.958 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:20.958 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:20.958 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:20.958 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:20.958 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:20.958 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:20.958 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:20.958 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:20.958 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:20.958 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:20.958 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c7bb46700 10 monclient: shutdown 2023-09-30T01:58:20.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c7bb46700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:20.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c7bb46700 1 -- 172.21.15.7:0/1639652431 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2c74096a50 msgr2=0x7f2c74132680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:20.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c7bb46700 1 --2- 172.21.15.7:0/1639652431 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2c74096a50 0x7f2c74132680 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2c64009f20 tx=0x7f2c64009840 comp rx=0 tx=0).stop 2023-09-30T01:58:20.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c7bb46700 1 -- 172.21.15.7:0/1639652431 shutdown_connections 2023-09-30T01:58:20.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c7bb46700 1 --2- 172.21.15.7:0/1639652431 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2c74096a50 0x7f2c74132680 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:20.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c7bb46700 1 --2- 172.21.15.7:0/1639652431 >> v2:172.21.15.7:3300/0 conn(0x7f2c74096e20 0x7f2c740971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:20.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.947+0000 7f2c7bb46700 1 -- 172.21.15.7:0/1639652431 >> 172.21.15.7:0/1639652431 conn(0x7f2c74070cf0 msgr2=0x7f2c740710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:20.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.948+0000 7f2c7bb46700 1 -- 172.21.15.7:0/1639652431 shutdown_connections 2023-09-30T01:58:20.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.948+0000 7f2c7bb46700 1 -- 172.21.15.7:0/1639652431 wait complete. 2023-09-30T01:58:20.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.948+0000 7f2c7bb46700 10 monclient: build_initial_monmap 2023-09-30T01:58:20.959 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.948+0000 7f2c7bb46700 10 monclient: monmap: 2023-09-30T01:58:20.959 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:20.959 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:58:20.960 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:58:20.960 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:58:20.960 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:20.960 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:20.960 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:58:20.960 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:58:20.960 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:58:20.960 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:20.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.948+0000 7f2c7bb46700 1 Processor -- start 2023-09-30T01:58:20.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.948+0000 7f2c7bb46700 1 -- start start 2023-09-30T01:58:20.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.948+0000 7f2c7bb46700 10 monclient: init 2023-09-30T01:58:20.960 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7bb46700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:20.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7bb46700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:58:20.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7bb46700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c74096a50 0x7f2c7412e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:20.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7bb46700 10 monclient(hunting): picked mon.noname-b con 0x7f2c74096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:58:20.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7bb46700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c74096e20 0x7f2c7409de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:20.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7bb46700 10 monclient(hunting): picked mon.noname-a con 0x7f2c74096e20 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:58:20.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7bb46700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2c7412ef00 0x7f2c7409e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:20.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7bb46700 10 monclient(hunting): picked mon.noname-c con 0x7f2c7412ef00 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:20.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7bb46700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:20.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7bb46700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c740a14b0 con 0x7f2c74096e20 2023-09-30T01:58:20.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7bb46700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:20.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7bb46700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c740a1790 con 0x7f2c74096a50 2023-09-30T01:58:20.961 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7bb46700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:20.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7bb46700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c740a1a70 con 0x7f2c7412ef00 2023-09-30T01:58:20.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7bb46700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:20.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7bb46700 10 monclient(hunting): authenticate will time out at 1390.014330s 2023-09-30T01:58:20.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c790e1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c74096e20 0x7f2c7409de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:20.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7a0e3700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2c7412ef00 0x7f2c7409e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:20.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c790e1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c74096e20 0x7f2c7409de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47616/0 (socket says 172.21.15.7:47616) 2023-09-30T01:58:20.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c790e1700 1 -- 172.21.15.7:0/2122806602 learned_addr learned my addr 172.21.15.7:0/2122806602 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:20.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c790e1700 10 monclient(hunting): get_auth_request con 0x7f2c74096e20 auth_method 0 2023-09-30T01:58:20.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c790e1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:20.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c790e1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:20.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c790e1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:20.962 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7a0e3700 10 monclient(hunting): get_auth_request con 0x7f2c7412ef00 auth_method 0 2023-09-30T01:58:20.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7a0e3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:20.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7a0e3700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:20.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7a0e3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:20.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c790e1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:20.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c790e1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:20.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c790e1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:20.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7a0e3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:20.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7a0e3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:20.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c7a0e3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:20.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c798e2700 1 --2- 172.21.15.7:0/2122806602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c74096a50 0x7f2c7412e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:20.963 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c798e2700 10 monclient(hunting): get_auth_request con 0x7f2c74096a50 auth_method 0 2023-09-30T01:58:20.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c798e2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:20.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c798e2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:20.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c798e2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:20.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c790e1700 10 monclient(hunting): handle_auth_done global_id 4599 payload 824 2023-09-30T01:58:20.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c790e1700 1 -- 172.21.15.7:0/2122806602 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2c7412ef00 msgr2=0x7f2c7409e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:20.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c790e1700 1 --2- 172.21.15.7:0/2122806602 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2c7412ef00 0x7f2c7409e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:20.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c790e1700 1 -- 172.21.15.7:0/2122806602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c74096a50 msgr2=0x7f2c7412e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:20.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c790e1700 1 --2- 172.21.15.7:0/2122806602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c74096a50 0x7f2c7412e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:20.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c790e1700 10 monclient: _finish_hunting 0 2023-09-30T01:58:20.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c790e1700 1 monclient: found mon.noname-a 2023-09-30T01:58:20.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c790e1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:20.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c790e1700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:58:20.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c790e1700 1 -- 172.21.15.7:0/2122806602 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c64009ae0 con 0x7f2c74096e20 2023-09-30T01:58:20.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.949+0000 7f2c798e2700 1 --2- 172.21.15.7:0/2122806602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c74096a50 0x7f2c7412e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:58:20.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.951+0000 7f2c790e1700 1 -- 172.21.15.7:0/2122806602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c74096e20 msgr2=0x7f2c7409de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:20.965 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.951+0000 7f2c790e1700 1 --2- 172.21.15.7:0/2122806602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c74096e20 0x7f2c7409de80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2c6800ec80 tx=0x7f2c680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:20.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.954+0000 7f2c71ffb700 1 -- 172.21.15.7:0/2122806602 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6800ca60 con 0x7f2c74096e20 2023-09-30T01:58:20.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.954+0000 7f2c71ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:20.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.954+0000 7f2c71ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:58:20.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.954+0000 7f2c71ffb700 10 monclient: dump: 2023-09-30T01:58:20.967 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:20.967 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:20.967 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:20.967 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:20.967 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:20.967 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:20.968 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:20.968 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:20.968 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:20.968 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:20.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.954+0000 7f2c71ffb700 10 monclient: _finish_auth 0 2023-09-30T01:58:20.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.954+0000 7f2c71ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:20.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.954+0000 7f2c7bb46700 5 monclient: authenticate success, global_id 4599 2023-09-30T01:58:20.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.954+0000 7f2c71ffb700 1 -- 172.21.15.7:0/2122806602 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c6800cbc0 con 0x7f2c74096e20 2023-09-30T01:58:20.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.954+0000 7f2c71ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:20.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.954+0000 7f2c71ffb700 1 -- 172.21.15.7:0/2122806602 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c68013470 con 0x7f2c74096e20 2023-09-30T01:58:20.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.954+0000 7f2c71ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:20.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.954+0000 7f2c71ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:58:20.968 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.954+0000 7f2c73fff700 4 set_mon_vals no callback set 2023-09-30T01:58:20.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.954+0000 7f2c71ffb700 10 monclient: dump: 2023-09-30T01:58:20.969 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:20.969 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:20.969 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:20.969 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:20.969 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:20.969 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:20.969 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:20.969 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:20.969 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:20.969 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:20.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.954+0000 7f2c7bb46700 10 monclient: _renew_subs 2023-09-30T01:58:20.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.954+0000 7f2c7bb46700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:20.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.954+0000 7f2c7bb46700 1 -- 172.21.15.7:0/2122806602 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c74142b40 con 0x7f2c74096e20 2023-09-30T01:58:20.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.954+0000 7f2c7bb46700 10 monclient: _renew_subs 2023-09-30T01:58:20.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.954+0000 7f2c7bb46700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:20.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.954+0000 7f2c7bb46700 1 -- 172.21.15.7:0/2122806602 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c740a05c0 con 0x7f2c74096e20 2023-09-30T01:58:20.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.955+0000 7f2c71ffb700 1 -- 172.21.15.7:0/2122806602 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f2c680137e0 con 0x7f2c74096e20 2023-09-30T01:58:20.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.955+0000 7f2c7bb46700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:20.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.955+0000 7f2c7bb46700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:20.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.955+0000 7f2c7bb46700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:20.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.955+0000 7f2c7bb46700 1 -- 172.21.15.7:0/2122806602 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c74132700 con 0x7f2c74096e20 2023-09-30T01:58:20.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.955+0000 7f2c71ffb700 1 --2- 172.21.15.7:0/2122806602 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2c60035830 0x7f2c60037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:20.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.955+0000 7f2c71ffb700 1 -- 172.21.15.7:0/2122806602 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f2c68048010 con 0x7f2c74096e20 2023-09-30T01:58:20.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.955+0000 7f2c798e2700 1 --2- 172.21.15.7:0/2122806602 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2c60035830 0x7f2c60037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:20.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.955+0000 7f2c798e2700 10 monclient: get_auth_request con 0x7f2c60035830 auth_method 0 2023-09-30T01:58:20.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.958+0000 7f2c798e2700 1 --2- 172.21.15.7:0/2122806602 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2c60035830 0x7f2c60037ce0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2c64009ab0 tx=0x7f2c64017000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:20.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.960+0000 7f2c71ffb700 1 -- 172.21.15.7:0/2122806602 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2c680795a0 con 0x7f2c74096e20 2023-09-30T01:58:20.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.960+0000 7f2c71ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:20.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:20.960+0000 7f2c71ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:58:21.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:21.194+0000 7f2c7bb46700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:58:21.194 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:21.194+0000 7f2c7bb46700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:21.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:21.194+0000 7f2c7bb46700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:21.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:21.194+0000 7f2c7bb46700 1 -- 172.21.15.7:0/2122806602 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2c741329e0 con 0x7f2c74096e20 2023-09-30T01:58:21.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:21.195+0000 7f2c71ffb700 1 -- 172.21.15.7:0/2122806602 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2c680236b0 con 0x7f2c74096e20 2023-09-30T01:58:21.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:21.195+0000 7f2c71ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:21.195 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:21.195+0000 7f2c71ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:58:21.195 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:58:21.196 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:21.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:21.196+0000 7f2c7bb46700 1 -- 172.21.15.7:0/2122806602 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2c60035830 msgr2=0x7f2c60037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:21.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:21.196+0000 7f2c7bb46700 1 --2- 172.21.15.7:0/2122806602 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2c60035830 0x7f2c60037ce0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2c64009ab0 tx=0x7f2c64017000 comp rx=0 tx=0).stop 2023-09-30T01:58:21.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:21.196+0000 7f2c7bb46700 10 monclient: shutdown 2023-09-30T01:58:21.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:21.196+0000 7f2c7bb46700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:21.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:21.196+0000 7f2c7bb46700 1 -- 172.21.15.7:0/2122806602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c74096e20 msgr2=0x7f2c7409de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:21.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:21.196+0000 7f2c7bb46700 1 --2- 172.21.15.7:0/2122806602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c74096e20 0x7f2c7409de80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2c6800ec80 tx=0x7f2c680091a0 comp rx=0 tx=0).stop 2023-09-30T01:58:21.197 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:21.196+0000 7f2c7bb46700 1 -- 172.21.15.7:0/2122806602 shutdown_connections 2023-09-30T01:58:21.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:21.196+0000 7f2c7bb46700 1 --2- 172.21.15.7:0/2122806602 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f2c60035830 0x7f2c60037ce0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:21.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:21.196+0000 7f2c7bb46700 1 --2- 172.21.15.7:0/2122806602 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2c7412ef00 0x7f2c7409e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:21.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:21.196+0000 7f2c7bb46700 1 --2- 172.21.15.7:0/2122806602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2c74096a50 0x7f2c7412e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:21.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:21.196+0000 7f2c7bb46700 1 --2- 172.21.15.7:0/2122806602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2c74096e20 0x7f2c7409de80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:21.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:21.197+0000 7f2c7bb46700 1 -- 172.21.15.7:0/2122806602 >> 172.21.15.7:0/2122806602 conn(0x7f2c74070cf0 msgr2=0x7f2c74131560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:21.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:21.197+0000 7f2c7bb46700 1 -- 172.21.15.7:0/2122806602 shutdown_connections 2023-09-30T01:58:21.198 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:21.197+0000 7f2c7bb46700 1 -- 172.21.15.7:0/2122806602 wait complete. 2023-09-30T01:58:24.209 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:24.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.336+0000 7fc40146e700 10 monclient: get_monmap_and_config 2023-09-30T01:58:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.336+0000 7fc40146e700 10 monclient: build_initial_monmap 2023-09-30T01:58:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.336+0000 7fc40146e700 10 monclient: monmap: 2023-09-30T01:58:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:58:24.337864+0000 2023-09-30T01:58:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:58:24.337864+0000 2023-09-30T01:58:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:24.340 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:24.341 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:58:24.341 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:58:24.341 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:58:24.341 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:58:24.341 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:58:24.341 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:24.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.337+0000 7fc40146e700 1 Processor -- start 2023-09-30T01:58:24.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.337+0000 7fc40146e700 1 -- start start 2023-09-30T01:58:24.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.337+0000 7fc40146e700 10 monclient: init 2023-09-30T01:58:24.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc40146e700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:24.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc40146e700 10 monclient: _add_conns ranks=[1,4,2,3,0] 2023-09-30T01:58:24.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc40146e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc3fc098540 addr v1:172.21.15.50:6789/0 2023-09-30T01:58:24.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc40146e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fc3fc1329e0 0x7fc3fc137400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:24.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc40146e700 10 monclient(hunting): picked mon.noname-c con 0x7fc3fc1329e0 addr v2:172.21.15.50:3300/0 2023-09-30T01:58:24.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc40146e700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc3fc137940 0x7fc3fc139d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:24.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc40146e700 10 monclient(hunting): picked mon.noname-b con 0x7fc3fc137940 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:24.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc40146e700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3fc069f10 con 0x7fc3fc098540 2023-09-30T01:58:24.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc40146e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:24.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc40146e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3fc0a0f40 con 0x7fc3fc1329e0 2023-09-30T01:58:24.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc40146e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:24.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc40146e700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3fc0a0790 con 0x7fc3fc137940 2023-09-30T01:58:24.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc40146e700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:24.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc40146e700 10 monclient(hunting): authenticate will time out at 1393.403261s 2023-09-30T01:58:24.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc400c6d700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc3fc137940 0x7fc3fc139d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:24.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc400c6d700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc3fc137940 0x7fc3fc139d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:56910/0 (socket says 172.21.15.7:56910) 2023-09-30T01:58:24.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc400c6d700 1 -- 172.21.15.7:0/3710564511 learned_addr learned my addr 172.21.15.7:0/3710564511 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:24.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc400c6d700 10 monclient(hunting): get_auth_request con 0x7fc3fc137940 auth_method 0 2023-09-30T01:58:24.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc400c6d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:24.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc400c6d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:24.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc400c6d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:24.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc400c6d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:24.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc400c6d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:24.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc3fb7fe700 1 --2- 172.21.15.7:0/3710564511 >> v2:172.21.15.50:3300/0 conn(0x7fc3fc1329e0 0x7fc3fc137400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:24.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc400c6d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:24.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc3fb7fe700 10 monclient(hunting): get_auth_request con 0x7fc3fc1329e0 auth_method 0 2023-09-30T01:58:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc3fb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc3fb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc3fb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc400c6d700 10 monclient(hunting): handle_auth_done global_id 4619 payload 274 2023-09-30T01:58:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc400c6d700 1 -- 172.21.15.7:0/3710564511 >> v2:172.21.15.50:3300/0 conn(0x7fc3fc1329e0 msgr2=0x7fc3fc137400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc400c6d700 1 --2- 172.21.15.7:0/3710564511 >> v2:172.21.15.50:3300/0 conn(0x7fc3fc1329e0 0x7fc3fc137400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc400c6d700 1 -- 172.21.15.7:0/3710564511 >> v1:172.21.15.50:6789/0 conn(0x7fc3fc098540 legacy=0x7fc3fc098910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc400c6d700 10 monclient: _finish_hunting 0 2023-09-30T01:58:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc400c6d700 1 monclient: found mon.noname-b 2023-09-30T01:58:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc400c6d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:24.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc400c6d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:58:24.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc400c6d700 1 -- 172.21.15.7:0/3710564511 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3fc096a50 con 0x7fc3fc137940 2023-09-30T01:58:24.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.338+0000 7fc3fb7fe700 1 --2- 172.21.15.7:0/3710564511 >> v2:172.21.15.50:3300/0 conn(0x7fc3fc1329e0 0x7fc3fc137400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:58:24.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.340+0000 7fc400c6d700 1 -- 172.21.15.7:0/3710564511 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc3fc137940 msgr2=0x7fc3fc139d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:24.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.340+0000 7fc400c6d700 1 --2- 172.21.15.7:0/3710564511 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc3fc137940 0x7fc3fc139d20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4009f20 tx=0x7fc3f4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c86d59e3e4f5b4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:24.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.340+0000 7fc3faffd700 1 -- 172.21.15.7:0/3710564511 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f400e070 con 0x7fc3fc137940 2023-09-30T01:58:24.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc3faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:24.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc3faffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:58:24.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc3faffd700 10 monclient: dump: 2023-09-30T01:58:24.351 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:24.351 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:24.351 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:24.351 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:24.352 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:24.352 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:24.352 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:24.352 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:24.352 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:24.352 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:24.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc3faffd700 10 monclient: _finish_auth 0 2023-09-30T01:58:24.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc3faffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:24.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc40146e700 5 monclient: authenticate success, global_id 4619 2023-09-30T01:58:24.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc3faffd700 1 -- 172.21.15.7:0/3710564511 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f400bc50 con 0x7fc3fc137940 2023-09-30T01:58:24.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc40146e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:58:24.352 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc3faffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:24.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc40146e700 10 monclient: get_monmap_and_config success 2023-09-30T01:58:24.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc3faffd700 1 -- 172.21.15.7:0/3710564511 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f40092e0 con 0x7fc3fc137940 2023-09-30T01:58:24.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc40146e700 4 set_mon_vals no callback set 2023-09-30T01:58:24.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc3faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:24.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc3faffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:58:24.353 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc3faffd700 10 monclient: dump: 2023-09-30T01:58:24.353 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:24.353 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:24.353 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:24.353 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:24.353 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:24.353 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:24.354 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:24.354 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:24.354 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:24.354 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:24.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc40146e700 10 monclient: shutdown 2023-09-30T01:58:24.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc40146e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:24.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc40146e700 1 -- 172.21.15.7:0/3710564511 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc3fc137940 msgr2=0x7fc3fc139d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:24.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc40146e700 1 --2- 172.21.15.7:0/3710564511 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc3fc137940 0x7fc3fc139d20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4009f20 tx=0x7fc3f4009840 comp rx=0 tx=0).stop 2023-09-30T01:58:24.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc40146e700 1 -- 172.21.15.7:0/3710564511 shutdown_connections 2023-09-30T01:58:24.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc40146e700 1 --2- 172.21.15.7:0/3710564511 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc3fc137940 0x7fc3fc139d20 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:24.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc40146e700 1 --2- 172.21.15.7:0/3710564511 >> v2:172.21.15.50:3300/0 conn(0x7fc3fc1329e0 0x7fc3fc137400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:24.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc40146e700 1 -- 172.21.15.7:0/3710564511 >> 172.21.15.7:0/3710564511 conn(0x7fc3fc071830 msgr2=0x7fc3fc0a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:24.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc40146e700 1 -- 172.21.15.7:0/3710564511 shutdown_connections 2023-09-30T01:58:24.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.341+0000 7fc40146e700 1 -- 172.21.15.7:0/3710564511 wait complete. 2023-09-30T01:58:24.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.342+0000 7fc40146e700 10 monclient: build_initial_monmap 2023-09-30T01:58:24.355 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.342+0000 7fc40146e700 10 monclient: monmap: 2023-09-30T01:58:24.355 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:24.355 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:58:24.355 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:58:24.355 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:58:24.355 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:24.355 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:24.355 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:58:24.356 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:58:24.356 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:58:24.356 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:24.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.342+0000 7fc40146e700 1 Processor -- start 2023-09-30T01:58:24.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.342+0000 7fc40146e700 1 -- start start 2023-09-30T01:58:24.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.342+0000 7fc40146e700 10 monclient: init 2023-09-30T01:58:24.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.342+0000 7fc40146e700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:24.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.342+0000 7fc40146e700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:58:24.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.342+0000 7fc40146e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3fc098540 0x7fc3fc134bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:24.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc40146e700 10 monclient(hunting): picked mon.noname-b con 0x7fc3fc098540 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:58:24.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc40146e700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc3fc1329e0 0x7fc3fc135100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:24.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc40146e700 10 monclient(hunting): picked mon.noname-c con 0x7fc3fc1329e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:24.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc40146e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3fc137940 0x7fc3fc135640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:24.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc40146e700 10 monclient(hunting): picked mon.noname-a con 0x7fc3fc137940 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:58:24.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc40146e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:24.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc40146e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3fc11cb50 con 0x7fc3fc137940 2023-09-30T01:58:24.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc40146e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:24.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc40146e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3fc11bc60 con 0x7fc3fc098540 2023-09-30T01:58:24.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc40146e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:24.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc40146e700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3fc137660 con 0x7fc3fc1329e0 2023-09-30T01:58:24.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc40146e700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:24.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc40146e700 10 monclient(hunting): authenticate will time out at 1393.408053s 2023-09-30T01:58:24.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fb7fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc3fc1329e0 0x7fc3fc135100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:24.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc400c6d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3fc137940 0x7fc3fc135640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:24.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fb7fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc3fc1329e0 0x7fc3fc135100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:56916/0 (socket says 172.21.15.7:56916) 2023-09-30T01:58:24.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fb7fe700 1 -- 172.21.15.7:0/2751219477 learned_addr learned my addr 172.21.15.7:0/2751219477 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:24.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc400c6d700 10 monclient(hunting): get_auth_request con 0x7fc3fc137940 auth_method 0 2023-09-30T01:58:24.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc400c6d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:24.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc400c6d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:24.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc400c6d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:24.358 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fb7fe700 10 monclient(hunting): get_auth_request con 0x7fc3fc1329e0 auth_method 0 2023-09-30T01:58:24.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:24.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:24.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:24.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:24.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:24.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:24.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc400c6d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:24.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc400c6d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:24.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc400c6d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:24.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fbfff700 1 --2- 172.21.15.7:0/2751219477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3fc098540 0x7fc3fc134bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:24.359 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fbfff700 10 monclient(hunting): get_auth_request con 0x7fc3fc098540 auth_method 0 2023-09-30T01:58:24.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:24.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:24.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:24.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fb7fe700 10 monclient(hunting): handle_auth_done global_id 4622 payload 824 2023-09-30T01:58:24.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fb7fe700 1 -- 172.21.15.7:0/2751219477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3fc098540 msgr2=0x7fc3fc134bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:24.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fb7fe700 1 --2- 172.21.15.7:0/2751219477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3fc098540 0x7fc3fc134bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:24.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fb7fe700 1 -- 172.21.15.7:0/2751219477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3fc137940 msgr2=0x7fc3fc135640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:24.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fb7fe700 1 --2- 172.21.15.7:0/2751219477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3fc137940 0x7fc3fc135640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:24.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fb7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:58:24.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fb7fe700 1 monclient: found mon.noname-c 2023-09-30T01:58:24.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:24.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fb7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:58:24.360 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fb7fe700 1 -- 172.21.15.7:0/2751219477 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f4009ae0 con 0x7fc3fc1329e0 2023-09-30T01:58:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc400c6d700 1 --2- 172.21.15.7:0/2751219477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3fc137940 0x7fc3fc135640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:58:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.343+0000 7fc3fbfff700 1 --2- 172.21.15.7:0/2751219477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3fc098540 0x7fc3fc134bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:58:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc3fb7fe700 1 -- 172.21.15.7:0/2751219477 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc3fc1329e0 msgr2=0x7fc3fc135100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc3fb7fe700 1 --2- 172.21.15.7:0/2751219477 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc3fc1329e0 0x7fc3fc135100 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400e8d0 tx=0x7fc3e4008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc3f3fff700 1 -- 172.21.15.7:0/2751219477 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4013620 con 0x7fc3fc1329e0 2023-09-30T01:58:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc3f3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc3f3fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:58:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc3f3fff700 10 monclient: dump: 2023-09-30T01:58:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:24.361 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:24.362 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:24.362 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:24.362 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:24.362 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:24.362 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:24.362 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:24.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc3f3fff700 10 monclient: _finish_auth 0 2023-09-30T01:58:24.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc3f3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:24.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc3f3fff700 1 -- 172.21.15.7:0/2751219477 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e4013780 con 0x7fc3fc1329e0 2023-09-30T01:58:24.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc40146e700 5 monclient: authenticate success, global_id 4622 2023-09-30T01:58:24.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc40146e700 10 monclient: _renew_subs 2023-09-30T01:58:24.362 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc40146e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:24.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc40146e700 1 -- 172.21.15.7:0/2751219477 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3fc0a05c0 con 0x7fc3fc1329e0 2023-09-30T01:58:24.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc3f3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:24.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc3f3fff700 1 -- 172.21.15.7:0/2751219477 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4013b20 con 0x7fc3fc1329e0 2023-09-30T01:58:24.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc3f3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:24.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc3f3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:58:24.363 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc3f3fff700 10 monclient: dump: 2023-09-30T01:58:24.363 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:24.363 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:24.363 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:24.363 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:24.363 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:24.363 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:24.364 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:24.364 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:24.364 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:24.364 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:24.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc3fa7fc700 4 set_mon_vals no callback set 2023-09-30T01:58:24.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc40146e700 10 monclient: _renew_subs 2023-09-30T01:58:24.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc40146e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:24.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.345+0000 7fc40146e700 1 -- 172.21.15.7:0/2751219477 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3fc136760 con 0x7fc3fc1329e0 2023-09-30T01:58:24.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.346+0000 7fc3f3fff700 1 -- 172.21.15.7:0/2751219477 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fc3e400c3a0 con 0x7fc3fc1329e0 2023-09-30T01:58:24.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.346+0000 7fc40146e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:24.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.346+0000 7fc40146e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:24.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.346+0000 7fc40146e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:24.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.346+0000 7fc40146e700 1 -- 172.21.15.7:0/2751219477 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3fc11ce90 con 0x7fc3fc1329e0 2023-09-30T01:58:24.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.346+0000 7fc3f3fff700 1 --2- 172.21.15.7:0/2751219477 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc3dc0357e0 0x7fc3dc037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:24.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.346+0000 7fc3f3fff700 1 -- 172.21.15.7:0/2751219477 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fc3e4047b50 con 0x7fc3fc1329e0 2023-09-30T01:58:24.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.347+0000 7fc3fbfff700 1 --2- 172.21.15.7:0/2751219477 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc3dc0357e0 0x7fc3dc037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:24.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.347+0000 7fc3fbfff700 10 monclient: get_auth_request con 0x7fc3dc0357e0 auth_method 0 2023-09-30T01:58:24.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.349+0000 7fc3fbfff700 1 --2- 172.21.15.7:0/2751219477 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc3dc0357e0 0x7fc3dc037c90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec007fd0 tx=0x7fc3ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:24.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.351+0000 7fc3f3fff700 1 -- 172.21.15.7:0/2751219477 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc3fc11ce90 con 0x7fc3fc1329e0 2023-09-30T01:58:24.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.351+0000 7fc3f3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:24.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.351+0000 7fc3f3fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:58:24.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.587+0000 7fc40146e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:58:24.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.587+0000 7fc40146e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:24.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.587+0000 7fc40146e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:24.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.587+0000 7fc40146e700 1 -- 172.21.15.7:0/2751219477 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc3fc11d070 con 0x7fc3fc1329e0 2023-09-30T01:58:24.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.587+0000 7fc3f3fff700 1 -- 172.21.15.7:0/2751219477 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc3fc11d070 con 0x7fc3fc1329e0 2023-09-30T01:58:24.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.587+0000 7fc3f3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:24.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.587+0000 7fc3f3fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:58:24.588 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:58:24.588 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:24.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.589+0000 7fc40146e700 1 -- 172.21.15.7:0/2751219477 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc3dc0357e0 msgr2=0x7fc3dc037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:24.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.589+0000 7fc40146e700 1 --2- 172.21.15.7:0/2751219477 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc3dc0357e0 0x7fc3dc037c90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec007fd0 tx=0x7fc3ec009040 comp rx=0 tx=0).stop 2023-09-30T01:58:24.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.589+0000 7fc40146e700 10 monclient: shutdown 2023-09-30T01:58:24.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.589+0000 7fc40146e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:24.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.589+0000 7fc40146e700 1 -- 172.21.15.7:0/2751219477 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc3fc1329e0 msgr2=0x7fc3fc135100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:24.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.589+0000 7fc40146e700 1 --2- 172.21.15.7:0/2751219477 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc3fc1329e0 0x7fc3fc135100 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400e8d0 tx=0x7fc3e4008040 comp rx=0 tx=0).stop 2023-09-30T01:58:24.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.589+0000 7fc40146e700 1 -- 172.21.15.7:0/2751219477 shutdown_connections 2023-09-30T01:58:24.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.589+0000 7fc40146e700 1 --2- 172.21.15.7:0/2751219477 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc3dc0357e0 0x7fc3dc037c90 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:24.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.589+0000 7fc40146e700 1 --2- 172.21.15.7:0/2751219477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3fc137940 0x7fc3fc135640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:24.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.589+0000 7fc40146e700 1 --2- 172.21.15.7:0/2751219477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3fc098540 0x7fc3fc134bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:24.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.589+0000 7fc40146e700 1 --2- 172.21.15.7:0/2751219477 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc3fc1329e0 0x7fc3fc135100 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:24.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.589+0000 7fc40146e700 1 -- 172.21.15.7:0/2751219477 >> 172.21.15.7:0/2751219477 conn(0x7fc3fc071830 msgr2=0x7fc3fc06a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:24.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.589+0000 7fc40146e700 1 -- 172.21.15.7:0/2751219477 shutdown_connections 2023-09-30T01:58:24.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:24.589+0000 7fc40146e700 1 -- 172.21.15.7:0/2751219477 wait complete. 2023-09-30T01:58:27.602 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:27.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.734+0000 7f096acbe700 10 monclient: get_monmap_and_config 2023-09-30T01:58:27.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.734+0000 7f096acbe700 10 monclient: build_initial_monmap 2023-09-30T01:58:27.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.734+0000 7f096acbe700 10 monclient: monmap: 2023-09-30T01:58:27.734 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:27.734 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:27.734 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:58:27.735212+0000 2023-09-30T01:58:27.734 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:58:27.735212+0000 2023-09-30T01:58:27.734 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:27.735 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:27.735 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:58:27.735 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:58:27.735 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:58:27.735 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:58:27.735 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:58:27.735 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:27.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.734+0000 7f096acbe700 1 Processor -- start 2023-09-30T01:58:27.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.734+0000 7f096acbe700 1 -- start start 2023-09-30T01:58:27.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.734+0000 7f096acbe700 10 monclient: init 2023-09-30T01:58:27.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.735+0000 7f096acbe700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:27.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.735+0000 7f096acbe700 10 monclient: _add_conns ranks=[1,0,4,3,2] 2023-09-30T01:58:27.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.735+0000 7f096acbe700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09640986e0 addr v1:172.21.15.50:6789/0 2023-09-30T01:58:27.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.735+0000 7f096acbe700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0964132660 addr v1:172.21.15.7:6789/0 2023-09-30T01:58:27.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.735+0000 7f096acbe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f0964132ae0 0x7f096413ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:27.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.735+0000 7f096acbe700 10 monclient(hunting): picked mon.noname-c con 0x7f0964132ae0 addr v2:172.21.15.50:3300/0 2023-09-30T01:58:27.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.735+0000 7f096acbe700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09640a0b70 con 0x7f0964132660 2023-09-30T01:58:27.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.735+0000 7f096acbe700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09640991c0 con 0x7f09640986e0 2023-09-30T01:58:27.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.735+0000 7f096acbe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:27.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.735+0000 7f096acbe700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0964096ed0 con 0x7f0964132ae0 2023-09-30T01:58:27.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.735+0000 7f096acbe700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:27.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.735+0000 7f096acbe700 10 monclient(hunting): authenticate will time out at 1396.800582s 2023-09-30T01:58:27.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.735+0000 7f0963fff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0964132660 0x7f0964137530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50290/0 (socket says 172.21.15.7:50290) 2023-09-30T01:58:27.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.735+0000 7f0963fff700 1 -- 172.21.15.7:0/3656594562 learned_addr learned my addr 172.21.15.7:0/3656594562 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:58:27.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.735+0000 7f096925b700 1 --2- 172.21.15.7:0/3656594562 >> v2:172.21.15.50:3300/0 conn(0x7f0964132ae0 0x7f096413ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:27.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.735+0000 7f096925b700 10 monclient(hunting): get_auth_request con 0x7f0964132ae0 auth_method 0 2023-09-30T01:58:27.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.735+0000 7f096925b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:27.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.735+0000 7f096925b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:27.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.735+0000 7f096925b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:27.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.735+0000 7f096925b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:27.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.736+0000 7f096925b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:27.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.736+0000 7f096925b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:27.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.736+0000 7f096925b700 10 monclient(hunting): handle_auth_done global_id 4537 payload 274 2023-09-30T01:58:27.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.736+0000 7f096925b700 1 -- 172.21.15.7:0/3656594562 >> v1:172.21.15.50:6789/0 conn(0x7f09640986e0 legacy=0x7f0964098ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:27.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.736+0000 7f096925b700 1 -- 172.21.15.7:0/3656594562 >> v1:172.21.15.7:6789/0 conn(0x7f0964132660 legacy=0x7f0964137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:27.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.736+0000 7f096925b700 10 monclient: _finish_hunting 0 2023-09-30T01:58:27.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.736+0000 7f096925b700 1 monclient: found mon.noname-c 2023-09-30T01:58:27.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.736+0000 7f096925b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:27.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.736+0000 7f096925b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3300/0 2023-09-30T01:58:27.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.736+0000 7f096925b700 1 -- 172.21.15.7:0/3656594562 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09640a0df0 con 0x7f0964132ae0 2023-09-30T01:58:27.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.738+0000 7f096925b700 1 -- 172.21.15.7:0/3656594562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0964132ae0 msgr2=0x7f096413ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:27.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.738+0000 7f096925b700 1 --2- 172.21.15.7:0/3656594562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0964132ae0 0x7f096413ac60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f095c009e60 tx=0x7f095c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe4b90062c8ecc5c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:27.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.738+0000 7f09637fe700 1 -- 172.21.15.7:0/3656594562 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095c00e070 con 0x7f0964132ae0 2023-09-30T01:58:27.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.738+0000 7f09637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:27.746 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.738+0000 7f09637fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:58:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.738+0000 7f09637fe700 10 monclient: dump: 2023-09-30T01:58:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:27.747 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.738+0000 7f09637fe700 10 monclient: _finish_auth 0 2023-09-30T01:58:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.738+0000 7f09637fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.738+0000 7f096acbe700 5 monclient: authenticate success, global_id 4537 2023-09-30T01:58:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.738+0000 7f09637fe700 1 -- 172.21.15.7:0/3656594562 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095c012430 con 0x7f0964132ae0 2023-09-30T01:58:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.738+0000 7f09637fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.738+0000 7f096acbe700 10 monclient: get_monmap_and_config success 2023-09-30T01:58:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.738+0000 7f09637fe700 1 -- 172.21.15.7:0/3656594562 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095c00bea0 con 0x7f0964132ae0 2023-09-30T01:58:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.738+0000 7f09637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.738+0000 7f096acbe700 4 set_mon_vals no callback set 2023-09-30T01:58:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.738+0000 7f09637fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:58:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.738+0000 7f09637fe700 10 monclient: dump: 2023-09-30T01:58:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:27.749 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:27.749 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:27.749 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:27.749 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:27.749 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:27.749 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:27.749 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:27.749 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:27.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.739+0000 7f096acbe700 10 monclient: shutdown 2023-09-30T01:58:27.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.739+0000 7f096acbe700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:27.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.739+0000 7f096acbe700 1 -- 172.21.15.7:0/3656594562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0964132ae0 msgr2=0x7f096413ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:27.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.739+0000 7f096acbe700 1 --2- 172.21.15.7:0/3656594562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0964132ae0 0x7f096413ac60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f095c009e60 tx=0x7f095c009780 comp rx=0 tx=0).stop 2023-09-30T01:58:27.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.739+0000 7f096acbe700 1 -- 172.21.15.7:0/3656594562 shutdown_connections 2023-09-30T01:58:27.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.739+0000 7f096acbe700 1 --2- 172.21.15.7:0/3656594562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0964132ae0 0x7f096413ac60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:27.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.739+0000 7f096acbe700 1 -- 172.21.15.7:0/3656594562 >> 172.21.15.7:0/3656594562 conn(0x7f0964070cf0 msgr2=0x7f09640710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:27.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.739+0000 7f096acbe700 1 -- 172.21.15.7:0/3656594562 shutdown_connections 2023-09-30T01:58:27.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.739+0000 7f096acbe700 1 -- 172.21.15.7:0/3656594562 wait complete. 2023-09-30T01:58:27.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.740+0000 7f096acbe700 10 monclient: build_initial_monmap 2023-09-30T01:58:27.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.740+0000 7f096acbe700 10 monclient: monmap: 2023-09-30T01:58:27.750 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:27.750 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:58:27.750 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:58:27.750 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:58:27.751 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:27.751 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:27.751 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:58:27.751 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:58:27.751 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:58:27.751 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:27.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.740+0000 7f096acbe700 1 Processor -- start 2023-09-30T01:58:27.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.740+0000 7f096acbe700 1 -- start start 2023-09-30T01:58:27.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.740+0000 7f096acbe700 10 monclient: init 2023-09-30T01:58:27.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096acbe700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:27.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096acbe700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:58:27.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096acbe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09640986e0 0x7f096412c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:27.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096acbe700 10 monclient(hunting): picked mon.noname-a con 0x7f09640986e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:58:27.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096acbe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0964132660 0x7f0964118df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:27.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096acbe700 10 monclient(hunting): picked mon.noname-b con 0x7f0964132660 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:58:27.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096acbe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0964132ae0 0x7f0964119330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:27.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096acbe700 10 monclient(hunting): picked mon.noname-c con 0x7f0964132ae0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:27.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096acbe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:27.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096acbe700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0964132270 con 0x7f09640986e0 2023-09-30T01:58:27.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096acbe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:27.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096acbe700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096409b910 con 0x7f0964132660 2023-09-30T01:58:27.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096acbe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:27.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096acbe700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f096411c4e0 con 0x7f0964132ae0 2023-09-30T01:58:27.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096acbe700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:27.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096acbe700 10 monclient(hunting): authenticate will time out at 1396.806144s 2023-09-30T01:58:27.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0968a5a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09640986e0 0x7f096412c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:27.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0968a5a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09640986e0 0x7f096412c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:47958/0 (socket says 172.21.15.7:47958) 2023-09-30T01:58:27.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0968a5a700 1 -- 172.21.15.7:0/1814533197 learned_addr learned my addr 172.21.15.7:0/1814533197 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:27.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0968a5a700 10 monclient(hunting): get_auth_request con 0x7f09640986e0 auth_method 0 2023-09-30T01:58:27.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0968a5a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:27.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0968a5a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:27.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0968a5a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:27.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096925b700 1 --2- 172.21.15.7:0/1814533197 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0964132ae0 0x7f0964119330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:27.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096925b700 10 monclient(hunting): get_auth_request con 0x7f0964132ae0 auth_method 0 2023-09-30T01:58:27.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096925b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:27.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096925b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:27.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096925b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:27.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0968a5a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:27.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0968a5a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:27.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0968a5a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:27.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096925b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:27.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096925b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:27.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096925b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:27.754 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0963fff700 1 --2- 172.21.15.7:0/1814533197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0964132660 0x7f0964118df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:27.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0968a5a700 10 monclient(hunting): handle_auth_done global_id 4605 payload 824 2023-09-30T01:58:27.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0968a5a700 1 -- 172.21.15.7:0/1814533197 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0964132ae0 msgr2=0x7f0964119330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:27.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0968a5a700 1 --2- 172.21.15.7:0/1814533197 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0964132ae0 0x7f0964119330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:27.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0968a5a700 1 -- 172.21.15.7:0/1814533197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0964132660 msgr2=0x7f0964118df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:27.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0968a5a700 1 --2- 172.21.15.7:0/1814533197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0964132660 0x7f0964118df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:27.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0968a5a700 10 monclient: _finish_hunting 0 2023-09-30T01:58:27.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0968a5a700 1 monclient: found mon.noname-a 2023-09-30T01:58:27.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0968a5a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:27.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0968a5a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:58:27.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0968a5a700 1 -- 172.21.15.7:0/1814533197 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095c009a20 con 0x7f09640986e0 2023-09-30T01:58:27.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0963fff700 10 monclient: get_auth_request con 0x7f0964132660 auth_method 0 2023-09-30T01:58:27.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f096925b700 1 --2- 172.21.15.7:0/1814533197 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0964132ae0 0x7f0964119330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:58:27.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.741+0000 7f0963fff700 1 --2- 172.21.15.7:0/1814533197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0964132660 0x7f0964118df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:58:27.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.743+0000 7f0968a5a700 1 -- 172.21.15.7:0/1814533197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09640986e0 msgr2=0x7f096412c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:27.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.743+0000 7f0968a5a700 1 --2- 172.21.15.7:0/1814533197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09640986e0 0x7f096412c500 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f095400c5d0 tx=0x7f0954007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:27.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.743+0000 7f0960ff9700 1 -- 172.21.15.7:0/1814533197 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09540117a0 con 0x7f09640986e0 2023-09-30T01:58:27.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.743+0000 7f0960ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:27.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.743+0000 7f0960ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:58:27.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.743+0000 7f0960ff9700 10 monclient: dump: 2023-09-30T01:58:27.756 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:27.756 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:27.757 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:27.757 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:27.757 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:27.757 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:27.757 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:27.757 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:27.757 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:27.757 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:27.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.743+0000 7f0960ff9700 10 monclient: _finish_auth 0 2023-09-30T01:58:27.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.743+0000 7f0960ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:27.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.743+0000 7f0960ff9700 1 -- 172.21.15.7:0/1814533197 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0954011900 con 0x7f09640986e0 2023-09-30T01:58:27.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.743+0000 7f096acbe700 5 monclient: authenticate success, global_id 4605 2023-09-30T01:58:27.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.744+0000 7f0960ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:27.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.744+0000 7f0960ff9700 1 -- 172.21.15.7:0/1814533197 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0954011cc0 con 0x7f09640986e0 2023-09-30T01:58:27.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.744+0000 7f0960ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:27.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.744+0000 7f0960ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:58:27.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.744+0000 7f0960ff9700 10 monclient: dump: 2023-09-30T01:58:27.758 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:27.758 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:27.758 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:27.758 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:27.758 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:27.759 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:27.759 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:27.759 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:27.759 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:27.759 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:27.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.744+0000 7f096acbe700 10 monclient: _renew_subs 2023-09-30T01:58:27.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.744+0000 7f096acbe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:27.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.744+0000 7f096acbe700 1 -- 172.21.15.7:0/1814533197 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f096411c820 con 0x7f09640986e0 2023-09-30T01:58:27.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.744+0000 7f0962ffd700 4 set_mon_vals no callback set 2023-09-30T01:58:27.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.744+0000 7f096acbe700 10 monclient: _renew_subs 2023-09-30T01:58:27.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.744+0000 7f096acbe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:27.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.744+0000 7f096acbe700 1 -- 172.21.15.7:0/1814533197 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f096412ba30 con 0x7f09640986e0 2023-09-30T01:58:27.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.744+0000 7f0960ff9700 1 -- 172.21.15.7:0/1814533197 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f0954012630 con 0x7f09640986e0 2023-09-30T01:58:27.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.744+0000 7f096acbe700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:27.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.745+0000 7f096acbe700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:27.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.745+0000 7f096acbe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:27.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.745+0000 7f096acbe700 1 -- 172.21.15.7:0/1814533197 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f096411d230 con 0x7f09640986e0 2023-09-30T01:58:27.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.745+0000 7f0960ff9700 1 --2- 172.21.15.7:0/1814533197 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f09500357e0 0x7f0950037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:27.762 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.745+0000 7f0960ff9700 1 -- 172.21.15.7:0/1814533197 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f09540479e0 con 0x7f09640986e0 2023-09-30T01:58:27.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.745+0000 7f0963fff700 1 --2- 172.21.15.7:0/1814533197 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f09500357e0 0x7f0950037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:27.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.745+0000 7f0963fff700 10 monclient: get_auth_request con 0x7f09500357e0 auth_method 0 2023-09-30T01:58:27.763 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.747+0000 7f0963fff700 1 --2- 172.21.15.7:0/1814533197 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f09500357e0 0x7f0950037c90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f096411a640 tx=0x7f0958009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:27.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.749+0000 7f0960ff9700 1 -- 172.21.15.7:0/1814533197 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f095401c460 con 0x7f09640986e0 2023-09-30T01:58:27.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.749+0000 7f0960ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:27.764 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.749+0000 7f0960ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:58:27.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.984+0000 7f096acbe700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:58:27.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.984+0000 7f096acbe700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:27.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.984+0000 7f096acbe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:27.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.984+0000 7f096acbe700 1 -- 172.21.15.7:0/1814533197 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0964064c10 con 0x7f09640986e0 2023-09-30T01:58:27.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.985+0000 7f0960ff9700 1 -- 172.21.15.7:0/1814533197 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f09540102e0 con 0x7f09640986e0 2023-09-30T01:58:27.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.985+0000 7f0960ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:27.985 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.985+0000 7f0960ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:58:27.986 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:58:27.986 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:27.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.987+0000 7f096acbe700 1 -- 172.21.15.7:0/1814533197 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f09500357e0 msgr2=0x7f0950037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:27.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.987+0000 7f096acbe700 1 --2- 172.21.15.7:0/1814533197 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f09500357e0 0x7f0950037c90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f096411a640 tx=0x7f0958009040 comp rx=0 tx=0).stop 2023-09-30T01:58:27.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.987+0000 7f096acbe700 10 monclient: shutdown 2023-09-30T01:58:27.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.987+0000 7f096acbe700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:27.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.987+0000 7f096acbe700 1 -- 172.21.15.7:0/1814533197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09640986e0 msgr2=0x7f096412c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:27.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.987+0000 7f096acbe700 1 --2- 172.21.15.7:0/1814533197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09640986e0 0x7f096412c500 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f095400c5d0 tx=0x7f0954007f60 comp rx=0 tx=0).stop 2023-09-30T01:58:27.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.987+0000 7f096acbe700 1 -- 172.21.15.7:0/1814533197 shutdown_connections 2023-09-30T01:58:27.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.987+0000 7f096acbe700 1 --2- 172.21.15.7:0/1814533197 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f09500357e0 0x7f0950037c90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:27.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.987+0000 7f096acbe700 1 --2- 172.21.15.7:0/1814533197 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09640986e0 0x7f096412c500 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:27.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.987+0000 7f096acbe700 1 --2- 172.21.15.7:0/1814533197 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0964132ae0 0x7f0964119330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:27.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.987+0000 7f096acbe700 1 --2- 172.21.15.7:0/1814533197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0964132660 0x7f0964118df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:27.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.987+0000 7f096acbe700 1 -- 172.21.15.7:0/1814533197 >> 172.21.15.7:0/1814533197 conn(0x7f0964070cf0 msgr2=0x7f096409f720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:27.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.987+0000 7f096acbe700 1 -- 172.21.15.7:0/1814533197 shutdown_connections 2023-09-30T01:58:27.988 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:27.987+0000 7f096acbe700 1 -- 172.21.15.7:0/1814533197 wait complete. 2023-09-30T01:58:31.000 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:31.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.128+0000 7f361fcc7700 10 monclient: get_monmap_and_config 2023-09-30T01:58:31.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.128+0000 7f361fcc7700 10 monclient: build_initial_monmap 2023-09-30T01:58:31.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.128+0000 7f361fcc7700 10 monclient: monmap: 2023-09-30T01:58:31.129 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:31.129 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:31.129 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:58:31.129684+0000 2023-09-30T01:58:31.129 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:58:31.129684+0000 2023-09-30T01:58:31.129 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:31.129 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:31.129 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:58:31.129 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:58:31.129 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:58:31.129 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:58:31.129 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:58:31.130 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:31.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.129+0000 7f361fcc7700 1 Processor -- start 2023-09-30T01:58:31.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.129+0000 7f361fcc7700 1 -- start start 2023-09-30T01:58:31.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.129+0000 7f361fcc7700 10 monclient: init 2023-09-30T01:58:31.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.129+0000 7f361fcc7700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:31.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.129+0000 7f361fcc7700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-09-30T01:58:31.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.129+0000 7f361fcc7700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3618096a50 0x7f36181327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:31.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.129+0000 7f361fcc7700 10 monclient(hunting): picked mon.noname-b con 0x7f3618096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:31.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.129+0000 7f361fcc7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f36180a2390 addr v1:172.21.15.50:6789/0 2023-09-30T01:58:31.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.129+0000 7f361fcc7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3618097530 addr v1:172.21.15.7:6789/0 2023-09-30T01:58:31.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.129+0000 7f361fcc7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36180980e0 con 0x7f3618097530 2023-09-30T01:58:31.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.129+0000 7f361fcc7700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36180991a0 con 0x7f36180a2390 2023-09-30T01:58:31.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.129+0000 7f361fcc7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:31.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.129+0000 7f361fcc7700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3618099480 con 0x7f3618096a50 2023-09-30T01:58:31.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.129+0000 7f361fcc7700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:31.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.129+0000 7f361fcc7700 10 monclient(hunting): authenticate will time out at 1400.194955s 2023-09-30T01:58:31.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.130+0000 7f361da63700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3618096a50 0x7f36181327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:31.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.130+0000 7f361e264700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3618097530 0x7f3618139720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50300/0 (socket says 172.21.15.7:50300) 2023-09-30T01:58:31.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.130+0000 7f361e264700 1 -- 172.21.15.7:0/4122240444 learned_addr learned my addr 172.21.15.7:0/4122240444 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:58:31.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.130+0000 7f361da63700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3618096a50 0x7f36181327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:56926/0 (socket says 172.21.15.7:56926) 2023-09-30T01:58:31.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.130+0000 7f361da63700 10 monclient(hunting): get_auth_request con 0x7f3618096a50 auth_method 0 2023-09-30T01:58:31.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.130+0000 7f361da63700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:31.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.130+0000 7f361da63700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:31.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.130+0000 7f361da63700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:31.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.130+0000 7f361da63700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:31.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.130+0000 7f361da63700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:31.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.130+0000 7f361da63700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:31.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.130+0000 7f361da63700 10 monclient(hunting): handle_auth_done global_id 4628 payload 274 2023-09-30T01:58:31.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.130+0000 7f361da63700 1 -- 172.21.15.7:0/4122240444 >> v1:172.21.15.50:6789/0 conn(0x7f36180a2390 legacy=0x7f3618096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:31.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.130+0000 7f361da63700 1 -- 172.21.15.7:0/4122240444 >> v1:172.21.15.7:6789/0 conn(0x7f3618097530 legacy=0x7f3618139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:31.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.130+0000 7f361da63700 10 monclient: _finish_hunting 0 2023-09-30T01:58:31.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.130+0000 7f361da63700 1 monclient: found mon.noname-b 2023-09-30T01:58:31.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.130+0000 7f361da63700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:31.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.130+0000 7f361da63700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:58:31.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.130+0000 7f361da63700 1 -- 172.21.15.7:0/4122240444 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3618097bf0 con 0x7f3618096a50 2023-09-30T01:58:31.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.132+0000 7f361da63700 1 -- 172.21.15.7:0/4122240444 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3618096a50 msgr2=0x7f36181327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:31.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.132+0000 7f361da63700 1 --2- 172.21.15.7:0/4122240444 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3618096a50 0x7f36181327a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3608009e90 tx=0x7f3608009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6649d205580273d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:31.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.132+0000 7f361ca61700 1 -- 172.21.15.7:0/4122240444 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360800e070 con 0x7f3618096a50 2023-09-30T01:58:31.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.132+0000 7f361ca61700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:31.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.132+0000 7f361ca61700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:58:31.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.132+0000 7f361ca61700 10 monclient: dump: 2023-09-30T01:58:31.141 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:31.142 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:31.142 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:31.142 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:31.142 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:31.142 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:31.142 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:31.142 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:31.142 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:31.142 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:31.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.132+0000 7f361ca61700 10 monclient: _finish_auth 0 2023-09-30T01:58:31.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.132+0000 7f361ca61700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:31.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.132+0000 7f361fcc7700 5 monclient: authenticate success, global_id 4628 2023-09-30T01:58:31.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.132+0000 7f361fcc7700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:58:31.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.132+0000 7f361ca61700 1 -- 172.21.15.7:0/4122240444 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3608012430 con 0x7f3618096a50 2023-09-30T01:58:31.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.132+0000 7f361ca61700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:31.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.132+0000 7f361fcc7700 10 monclient: get_monmap_and_config success 2023-09-30T01:58:31.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.132+0000 7f361fcc7700 4 set_mon_vals no callback set 2023-09-30T01:58:31.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.132+0000 7f361ca61700 1 -- 172.21.15.7:0/4122240444 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36080127d0 con 0x7f3618096a50 2023-09-30T01:58:31.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.132+0000 7f361ca61700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:31.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.132+0000 7f361ca61700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:58:31.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.132+0000 7f361ca61700 10 monclient: dump: 2023-09-30T01:58:31.143 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:31.143 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:31.144 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:31.144 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:31.144 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:31.144 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:31.144 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:31.145 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:31.145 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:31.145 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:31.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.133+0000 7f361fcc7700 10 monclient: shutdown 2023-09-30T01:58:31.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.133+0000 7f361fcc7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:31.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.133+0000 7f361fcc7700 1 -- 172.21.15.7:0/4122240444 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3618096a50 msgr2=0x7f36181327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:31.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.133+0000 7f361fcc7700 1 --2- 172.21.15.7:0/4122240444 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3618096a50 0x7f36181327a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3608009e90 tx=0x7f3608009780 comp rx=0 tx=0).stop 2023-09-30T01:58:31.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.133+0000 7f361fcc7700 1 -- 172.21.15.7:0/4122240444 shutdown_connections 2023-09-30T01:58:31.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.133+0000 7f361fcc7700 1 --2- 172.21.15.7:0/4122240444 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3618096a50 0x7f36181327a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:31.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.133+0000 7f361fcc7700 1 -- 172.21.15.7:0/4122240444 >> 172.21.15.7:0/4122240444 conn(0x7f3618070cf0 msgr2=0x7f36180710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:31.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.133+0000 7f361fcc7700 1 -- 172.21.15.7:0/4122240444 shutdown_connections 2023-09-30T01:58:31.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.133+0000 7f361fcc7700 1 -- 172.21.15.7:0/4122240444 wait complete. 2023-09-30T01:58:31.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.133+0000 7f361fcc7700 10 monclient: build_initial_monmap 2023-09-30T01:58:31.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.133+0000 7f361fcc7700 10 monclient: monmap: 2023-09-30T01:58:31.146 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:31.146 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:58:31.146 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:58:31.146 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:58:31.146 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:31.146 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:31.146 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:58:31.146 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:58:31.146 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:58:31.147 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:31.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361fcc7700 1 Processor -- start 2023-09-30T01:58:31.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361fcc7700 1 -- start start 2023-09-30T01:58:31.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361fcc7700 10 monclient: init 2023-09-30T01:58:31.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361fcc7700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:31.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361fcc7700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:58:31.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361fcc7700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3618096a50 0x7f361809bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:31.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361fcc7700 10 monclient(hunting): picked mon.noname-c con 0x7f3618096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:31.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361fcc7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3618097530 0x7f361809bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:31.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361fcc7700 10 monclient(hunting): picked mon.noname-b con 0x7f3618097530 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:58:31.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361fcc7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36180a2390 0x7f361809c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:31.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361fcc7700 10 monclient(hunting): picked mon.noname-a con 0x7f36180a2390 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:58:31.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361fcc7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:31.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361fcc7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361809f1c0 con 0x7f36180a2390 2023-09-30T01:58:31.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361fcc7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:31.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361fcc7700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361809e0f0 con 0x7f3618097530 2023-09-30T01:58:31.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361fcc7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:31.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361fcc7700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3618133550 con 0x7f3618096a50 2023-09-30T01:58:31.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361fcc7700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:31.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361fcc7700 10 monclient(hunting): authenticate will time out at 1400.199780s 2023-09-30T01:58:31.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361da63700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3618096a50 0x7f361809bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:31.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361e264700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36180a2390 0x7f361809c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:31.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361da63700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3618096a50 0x7f361809bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:56936/0 (socket says 172.21.15.7:56936) 2023-09-30T01:58:31.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361da63700 1 -- 172.21.15.7:0/1641389768 learned_addr learned my addr 172.21.15.7:0/1641389768 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:31.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361da63700 10 monclient(hunting): get_auth_request con 0x7f3618096a50 auth_method 0 2023-09-30T01:58:31.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361da63700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:31.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361da63700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:31.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361da63700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:31.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361e264700 10 monclient(hunting): get_auth_request con 0x7f36180a2390 auth_method 0 2023-09-30T01:58:31.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361e264700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:31.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361e264700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:31.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361e264700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:31.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361da63700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:31.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361da63700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:31.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361da63700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:31.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361d262700 1 --2- 172.21.15.7:0/1641389768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3618097530 0x7f361809bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:31.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361e264700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:31.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.134+0000 7f361e264700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:31.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.135+0000 7f361e264700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:31.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.135+0000 7f361d262700 10 monclient(hunting): get_auth_request con 0x7f3618097530 auth_method 0 2023-09-30T01:58:31.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.135+0000 7f361d262700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:31.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.135+0000 7f361d262700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:31.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.135+0000 7f361d262700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:31.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.135+0000 7f361da63700 10 monclient(hunting): handle_auth_done global_id 4631 payload 824 2023-09-30T01:58:31.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.135+0000 7f361da63700 1 -- 172.21.15.7:0/1641389768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3618097530 msgr2=0x7f361809bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:31.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.135+0000 7f361da63700 1 --2- 172.21.15.7:0/1641389768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3618097530 0x7f361809bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:31.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.135+0000 7f361da63700 1 -- 172.21.15.7:0/1641389768 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36180a2390 msgr2=0x7f361809c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:31.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.135+0000 7f361da63700 1 --2- 172.21.15.7:0/1641389768 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36180a2390 0x7f361809c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:31.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.135+0000 7f361da63700 10 monclient: _finish_hunting 0 2023-09-30T01:58:31.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.135+0000 7f361da63700 1 monclient: found mon.noname-c 2023-09-30T01:58:31.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.135+0000 7f361da63700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:31.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.135+0000 7f361da63700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:58:31.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.135+0000 7f361da63700 1 -- 172.21.15.7:0/1641389768 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3608009a20 con 0x7f3618096a50 2023-09-30T01:58:31.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.135+0000 7f361d262700 1 --2- 172.21.15.7:0/1641389768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3618097530 0x7f361809bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:58:31.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.135+0000 7f361e264700 1 --2- 172.21.15.7:0/1641389768 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36180a2390 0x7f361809c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:58:31.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f361da63700 1 -- 172.21.15.7:0/1641389768 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3618096a50 msgr2=0x7f361809bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:31.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f361da63700 1 --2- 172.21.15.7:0/1641389768 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3618096a50 0x7f361809bab0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3608003000 tx=0x7f3608018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:31.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f3615ffb700 1 -- 172.21.15.7:0/1641389768 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608012dd0 con 0x7f3618096a50 2023-09-30T01:58:31.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f3615ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:31.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f3615ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:58:31.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f3615ffb700 10 monclient: dump: 2023-09-30T01:58:31.152 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:31.152 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:31.153 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:31.153 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:31.153 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:31.153 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:31.153 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:31.153 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:31.153 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:31.153 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:31.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f3615ffb700 10 monclient: _finish_auth 0 2023-09-30T01:58:31.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f3615ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:31.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f361fcc7700 5 monclient: authenticate success, global_id 4631 2023-09-30T01:58:31.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f3615ffb700 1 -- 172.21.15.7:0/1641389768 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3608021070 con 0x7f3618096a50 2023-09-30T01:58:31.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f3615ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:31.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f3615ffb700 1 -- 172.21.15.7:0/1641389768 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608026680 con 0x7f3618096a50 2023-09-30T01:58:31.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f361fcc7700 10 monclient: _renew_subs 2023-09-30T01:58:31.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f361fcc7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:31.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f361fcc7700 1 -- 172.21.15.7:0/1641389768 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f361813ac30 con 0x7f3618096a50 2023-09-30T01:58:31.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f3617fff700 4 set_mon_vals no callback set 2023-09-30T01:58:31.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f3615ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:31.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f3615ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:58:31.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f3615ffb700 10 monclient: dump: 2023-09-30T01:58:31.154 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:31.154 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:31.155 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:31.155 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:31.155 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:31.155 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:31.155 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:31.155 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:31.155 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:31.155 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:31.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f361fcc7700 10 monclient: _renew_subs 2023-09-30T01:58:31.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f361fcc7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:31.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.137+0000 7f361fcc7700 1 -- 172.21.15.7:0/1641389768 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36180a0000 con 0x7f3618096a50 2023-09-30T01:58:31.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.138+0000 7f3615ffb700 1 -- 172.21.15.7:0/1641389768 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f3608023030 con 0x7f3618096a50 2023-09-30T01:58:31.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.138+0000 7f361fcc7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:31.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.138+0000 7f361fcc7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:31.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.138+0000 7f361fcc7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:31.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.138+0000 7f361fcc7700 1 -- 172.21.15.7:0/1641389768 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3618064c10 con 0x7f3618096a50 2023-09-30T01:58:31.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.138+0000 7f3615ffb700 1 --2- 172.21.15.7:0/1641389768 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f36040357e0 0x7f3604037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:31.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.138+0000 7f3615ffb700 1 -- 172.21.15.7:0/1641389768 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f3608049f90 con 0x7f3618096a50 2023-09-30T01:58:31.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.139+0000 7f361d262700 1 --2- 172.21.15.7:0/1641389768 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f36040357e0 0x7f3604037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:31.156 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.139+0000 7f361d262700 10 monclient: get_auth_request con 0x7f36040357e0 auth_method 0 2023-09-30T01:58:31.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.141+0000 7f361d262700 1 --2- 172.21.15.7:0/1641389768 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f36040357e0 0x7f3604037c90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f360c007fd0 tx=0x7f360c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:31.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.143+0000 7f3615ffb700 1 -- 172.21.15.7:0/1641389768 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f360802f440 con 0x7f3618096a50 2023-09-30T01:58:31.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.143+0000 7f3615ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:31.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.143+0000 7f3615ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:58:31.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.380+0000 7f361fcc7700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:58:31.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.380+0000 7f361fcc7700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:31.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.380+0000 7f361fcc7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:31.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.380+0000 7f361fcc7700 1 -- 172.21.15.7:0/1641389768 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3618064c10 con 0x7f3618096a50 2023-09-30T01:58:31.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.380+0000 7f3615ffb700 1 -- 172.21.15.7:0/1641389768 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f360801b3a0 con 0x7f3618096a50 2023-09-30T01:58:31.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.380+0000 7f3615ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:31.381 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.380+0000 7f3615ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:58:31.381 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:58:31.381 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.382+0000 7f361fcc7700 1 -- 172.21.15.7:0/1641389768 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f36040357e0 msgr2=0x7f3604037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.382+0000 7f361fcc7700 1 --2- 172.21.15.7:0/1641389768 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f36040357e0 0x7f3604037c90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f360c007fd0 tx=0x7f360c009040 comp rx=0 tx=0).stop 2023-09-30T01:58:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.382+0000 7f361fcc7700 10 monclient: shutdown 2023-09-30T01:58:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.382+0000 7f361fcc7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.382+0000 7f361fcc7700 1 -- 172.21.15.7:0/1641389768 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3618096a50 msgr2=0x7f361809bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.382+0000 7f361fcc7700 1 --2- 172.21.15.7:0/1641389768 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3618096a50 0x7f361809bab0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3608003000 tx=0x7f3608018000 comp rx=0 tx=0).stop 2023-09-30T01:58:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.382+0000 7f361fcc7700 1 -- 172.21.15.7:0/1641389768 shutdown_connections 2023-09-30T01:58:31.383 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.382+0000 7f361fcc7700 1 --2- 172.21.15.7:0/1641389768 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f36040357e0 0x7f3604037c90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.382+0000 7f361fcc7700 1 --2- 172.21.15.7:0/1641389768 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f36180a2390 0x7f361809c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.382+0000 7f361fcc7700 1 --2- 172.21.15.7:0/1641389768 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3618096a50 0x7f361809bab0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.382+0000 7f361fcc7700 1 --2- 172.21.15.7:0/1641389768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3618097530 0x7f361809bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.382+0000 7f361fcc7700 1 -- 172.21.15.7:0/1641389768 >> 172.21.15.7:0/1641389768 conn(0x7f3618070cf0 msgr2=0x7f36180a4500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.382+0000 7f361fcc7700 1 -- 172.21.15.7:0/1641389768 shutdown_connections 2023-09-30T01:58:31.384 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:31.382+0000 7f361fcc7700 1 -- 172.21.15.7:0/1641389768 wait complete. 2023-09-30T01:58:34.395 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:34.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.525+0000 7f8529f49700 10 monclient: get_monmap_and_config 2023-09-30T01:58:34.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.525+0000 7f8529f49700 10 monclient: build_initial_monmap 2023-09-30T01:58:34.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.525+0000 7f8529f49700 10 monclient: monmap: 2023-09-30T01:58:34.526 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:34.526 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:34.526 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:58:34.526371+0000 2023-09-30T01:58:34.527 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:58:34.526371+0000 2023-09-30T01:58:34.527 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:34.527 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:34.527 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:58:34.527 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:58:34.528 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:58:34.528 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:58:34.528 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:58:34.528 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:34.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8529f49700 1 Processor -- start 2023-09-30T01:58:34.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8529f49700 1 -- start start 2023-09-30T01:58:34.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8529f49700 10 monclient: init 2023-09-30T01:58:34.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8529f49700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:34.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8529f49700 10 monclient: _add_conns ranks=[3,4,2,0,1] 2023-09-30T01:58:34.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8529f49700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f85240a29e0 0x7f8524096a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:34.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8529f49700 10 monclient(hunting): picked mon.noname-a con 0x7f85240a29e0 addr v2:172.21.15.7:3300/0 2023-09-30T01:58:34.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8529f49700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8524096f90 0x7f8524097390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:34.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8529f49700 10 monclient(hunting): picked mon.noname-c con 0x7f8524096f90 addr v2:172.21.15.50:3300/0 2023-09-30T01:58:34.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8529f49700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8524132220 0x7f85241326a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:34.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8529f49700 10 monclient(hunting): picked mon.noname-b con 0x7f8524132220 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:34.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8529f49700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:34.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8529f49700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8524098ea0 con 0x7f85240a29e0 2023-09-30T01:58:34.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8529f49700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:34.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8529f49700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85240978d0 con 0x7f8524096f90 2023-09-30T01:58:34.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8529f49700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:34.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8529f49700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8524069cc0 con 0x7f8524132220 2023-09-30T01:58:34.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8529f49700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:34.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8529f49700 10 monclient(hunting): authenticate will time out at 1403.591733s 2023-09-30T01:58:34.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f85237fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f85240a29e0 0x7f8524096a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:34.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f85237fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f85240a29e0 0x7f8524096a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44802/0 (socket says 172.21.15.7:44802) 2023-09-30T01:58:34.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8523fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8524132220 0x7f85241326a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:34.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f85237fe700 1 -- 172.21.15.7:0/3108053646 learned_addr learned my addr 172.21.15.7:0/3108053646 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:34.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f85237fe700 10 monclient(hunting): get_auth_request con 0x7f85240a29e0 auth_method 0 2023-09-30T01:58:34.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f85237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:34.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f85237fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:34.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f85237fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:34.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8523fff700 10 monclient(hunting): get_auth_request con 0x7f8524132220 auth_method 0 2023-09-30T01:58:34.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8523fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:34.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8523fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:34.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f8523fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:34.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f85237fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:34.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f85237fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:34.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.526+0000 7f85237fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:34.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.527+0000 7f8523fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:34.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.527+0000 7f8523fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:34.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.527+0000 7f851affd700 1 --2- 172.21.15.7:0/3108053646 >> v2:172.21.15.50:3300/0 conn(0x7f8524096f90 0x7f8524097390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:34.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.527+0000 7f8523fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.527+0000 7f851affd700 10 monclient(hunting): get_auth_request con 0x7f8524096f90 auth_method 0 2023-09-30T01:58:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.527+0000 7f851affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.527+0000 7f851affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.527+0000 7f851affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.527+0000 7f85237fe700 10 monclient(hunting): handle_auth_done global_id 4611 payload 274 2023-09-30T01:58:34.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.527+0000 7f85237fe700 1 -- 172.21.15.7:0/3108053646 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8524132220 msgr2=0x7f85241326a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:34.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.527+0000 7f85237fe700 1 --2- 172.21.15.7:0/3108053646 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8524132220 0x7f85241326a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:34.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.527+0000 7f85237fe700 1 -- 172.21.15.7:0/3108053646 >> v2:172.21.15.50:3300/0 conn(0x7f8524096f90 msgr2=0x7f8524097390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:34.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.527+0000 7f85237fe700 1 --2- 172.21.15.7:0/3108053646 >> v2:172.21.15.50:3300/0 conn(0x7f8524096f90 0x7f8524097390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:34.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.527+0000 7f85237fe700 10 monclient: _finish_hunting 0 2023-09-30T01:58:34.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.527+0000 7f85237fe700 1 monclient: found mon.noname-a 2023-09-30T01:58:34.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.527+0000 7f85237fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.527+0000 7f85237fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:58:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.527+0000 7f85237fe700 1 -- 172.21.15.7:0/3108053646 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85240a2db0 con 0x7f85240a29e0 2023-09-30T01:58:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.529+0000 7f85237fe700 1 -- 172.21.15.7:0/3108053646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85240a29e0 msgr2=0x7f8524096a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.529+0000 7f85237fe700 1 --2- 172.21.15.7:0/3108053646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85240a29e0 0x7f8524096a50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8524098070 tx=0x7f8514009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=817de616db24565f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.529+0000 7f8522ffd700 1 -- 172.21.15.7:0/3108053646 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8514009e80 con 0x7f85240a29e0 2023-09-30T01:58:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.529+0000 7f8522ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.529+0000 7f8522ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:58:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.529+0000 7f8522ffd700 10 monclient: dump: 2023-09-30T01:58:34.554 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:34.555 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:34.556 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.529+0000 7f8522ffd700 10 monclient: _finish_auth 0 2023-09-30T01:58:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.529+0000 7f8522ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.529+0000 7f8529f49700 5 monclient: authenticate success, global_id 4611 2023-09-30T01:58:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.529+0000 7f8529f49700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:58:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.529+0000 7f8522ffd700 1 -- 172.21.15.7:0/3108053646 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8514005e20 con 0x7f85240a29e0 2023-09-30T01:58:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.529+0000 7f8522ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:34.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.529+0000 7f8529f49700 10 monclient: get_monmap_and_config success 2023-09-30T01:58:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.529+0000 7f8529f49700 4 set_mon_vals no callback set 2023-09-30T01:58:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.529+0000 7f8522ffd700 1 -- 172.21.15.7:0/3108053646 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851400b840 con 0x7f85240a29e0 2023-09-30T01:58:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.529+0000 7f8522ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.529+0000 7f8522ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:58:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.529+0000 7f8522ffd700 10 monclient: dump: 2023-09-30T01:58:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:34.557 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:34.558 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:34.558 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:34.558 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:34.558 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:34.558 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:34.558 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:34.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.530+0000 7f8529f49700 10 monclient: shutdown 2023-09-30T01:58:34.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.530+0000 7f8529f49700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:34.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.530+0000 7f8529f49700 1 -- 172.21.15.7:0/3108053646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85240a29e0 msgr2=0x7f8524096a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:34.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.530+0000 7f8529f49700 1 --2- 172.21.15.7:0/3108053646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85240a29e0 0x7f8524096a50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8524098070 tx=0x7f8514009220 comp rx=0 tx=0).stop 2023-09-30T01:58:34.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.530+0000 7f8529f49700 1 -- 172.21.15.7:0/3108053646 shutdown_connections 2023-09-30T01:58:34.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.530+0000 7f8529f49700 1 --2- 172.21.15.7:0/3108053646 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8524132220 0x7f85241326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:34.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.530+0000 7f8529f49700 1 --2- 172.21.15.7:0/3108053646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85240a29e0 0x7f8524096a50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:34.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.530+0000 7f8529f49700 1 --2- 172.21.15.7:0/3108053646 >> v2:172.21.15.50:3300/0 conn(0x7f8524096f90 0x7f8524097390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:34.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.530+0000 7f8529f49700 1 -- 172.21.15.7:0/3108053646 >> 172.21.15.7:0/3108053646 conn(0x7f85240710d0 msgr2=0x7f85240714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:34.559 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.530+0000 7f8529f49700 1 -- 172.21.15.7:0/3108053646 shutdown_connections 2023-09-30T01:58:34.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.530+0000 7f8529f49700 1 -- 172.21.15.7:0/3108053646 wait complete. 2023-09-30T01:58:34.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.530+0000 7f8529f49700 10 monclient: build_initial_monmap 2023-09-30T01:58:34.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.530+0000 7f8529f49700 10 monclient: monmap: 2023-09-30T01:58:34.560 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:34.560 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:58:34.560 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:58:34.560 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:58:34.560 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:34.560 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:34.561 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:58:34.561 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:58:34.561 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:58:34.561 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:34.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f8529f49700 1 Processor -- start 2023-09-30T01:58:34.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f8529f49700 1 -- start start 2023-09-30T01:58:34.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f8529f49700 10 monclient: init 2023-09-30T01:58:34.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f8529f49700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:34.561 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f8529f49700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:58:34.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f8529f49700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8524096f90 0x7f85241419e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:34.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f8529f49700 10 monclient(hunting): picked mon.noname-c con 0x7f8524096f90 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:34.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f8529f49700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85240a29e0 0x7f8524141f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:34.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f8529f49700 10 monclient(hunting): picked mon.noname-a con 0x7f85240a29e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:58:34.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f8529f49700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8524132220 0x7f852411cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:34.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f8529f49700 10 monclient(hunting): picked mon.noname-b con 0x7f8524132220 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:58:34.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f8529f49700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:34.562 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f8529f49700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8524120420 con 0x7f85240a29e0 2023-09-30T01:58:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f8529f49700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f8529f49700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85240a1220 con 0x7f8524132220 2023-09-30T01:58:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f8529f49700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f8529f49700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85240a5220 con 0x7f8524096f90 2023-09-30T01:58:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f8529f49700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f8529f49700 10 monclient(hunting): authenticate will time out at 1403.596731s 2023-09-30T01:58:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f851affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85240a29e0 0x7f8524141f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:34.563 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f85237fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8524096f90 0x7f85241419e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:34.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f851affd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85240a29e0 0x7f8524141f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44804/0 (socket says 172.21.15.7:44804) 2023-09-30T01:58:34.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f85237fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8524096f90 0x7f85241419e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:51268/0 (socket says 172.21.15.7:51268) 2023-09-30T01:58:34.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f851affd700 1 -- 172.21.15.7:0/62759190 learned_addr learned my addr 172.21.15.7:0/62759190 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:34.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f851affd700 10 monclient(hunting): get_auth_request con 0x7f85240a29e0 auth_method 0 2023-09-30T01:58:34.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f851affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:34.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f851affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:34.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f851affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:34.564 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f85237fe700 10 monclient(hunting): get_auth_request con 0x7f8524096f90 auth_method 0 2023-09-30T01:58:34.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f85237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:34.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f85237fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:34.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f85237fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:34.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f851affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:34.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f851affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:34.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f851affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:34.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f85237fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:34.565 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f85237fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:34.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.531+0000 7f85237fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:34.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.532+0000 7f8523fff700 1 --2- 172.21.15.7:0/62759190 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8524132220 0x7f852411cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:34.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.532+0000 7f851affd700 10 monclient(hunting): handle_auth_done global_id 4614 payload 824 2023-09-30T01:58:34.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.532+0000 7f851affd700 1 -- 172.21.15.7:0/62759190 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8524096f90 msgr2=0x7f85241419e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:34.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.532+0000 7f851affd700 1 --2- 172.21.15.7:0/62759190 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8524096f90 0x7f85241419e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:34.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.532+0000 7f851affd700 1 -- 172.21.15.7:0/62759190 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8524132220 msgr2=0x7f852411cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:34.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.532+0000 7f851affd700 1 --2- 172.21.15.7:0/62759190 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8524132220 0x7f852411cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:34.566 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.532+0000 7f851affd700 10 monclient: _finish_hunting 0 2023-09-30T01:58:34.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.532+0000 7f851affd700 1 monclient: found mon.noname-a 2023-09-30T01:58:34.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.532+0000 7f851affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:34.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.532+0000 7f851affd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:58:34.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.532+0000 7f851affd700 1 -- 172.21.15.7:0/62759190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8514009480 con 0x7f85240a29e0 2023-09-30T01:58:34.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.532+0000 7f8523fff700 10 monclient: get_auth_request con 0x7f8524132220 auth_method 0 2023-09-30T01:58:34.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.532+0000 7f85237fe700 1 --2- 172.21.15.7:0/62759190 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8524096f90 0x7f85241419e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:58:34.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.532+0000 7f8523fff700 1 --2- 172.21.15.7:0/62759190 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8524132220 0x7f852411cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:58:34.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f851affd700 1 -- 172.21.15.7:0/62759190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85240a29e0 msgr2=0x7f8524141f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:34.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f851affd700 1 --2- 172.21.15.7:0/62759190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85240a29e0 0x7f8524141f20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f850c00d890 tx=0x7f850c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:34.567 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f851bfff700 1 -- 172.21.15.7:0/62759190 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c012800 con 0x7f85240a29e0 2023-09-30T01:58:34.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f851bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:34.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f851bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:58:34.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f851bfff700 10 monclient: dump: 2023-09-30T01:58:34.568 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:34.568 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:34.568 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:34.568 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:34.568 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:34.568 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:34.569 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:34.569 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:34.569 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:34.569 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:34.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f851bfff700 10 monclient: _finish_auth 0 2023-09-30T01:58:34.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f851bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:34.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f8529f49700 5 monclient: authenticate success, global_id 4614 2023-09-30T01:58:34.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f8529f49700 10 monclient: _renew_subs 2023-09-30T01:58:34.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f851bfff700 1 -- 172.21.15.7:0/62759190 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850c012960 con 0x7f85240a29e0 2023-09-30T01:58:34.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f8529f49700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:34.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f8529f49700 1 -- 172.21.15.7:0/62759190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8524120140 con 0x7f85240a29e0 2023-09-30T01:58:34.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f851bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:34.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f851bfff700 1 -- 172.21.15.7:0/62759190 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c012ce0 con 0x7f85240a29e0 2023-09-30T01:58:34.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f851bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:34.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f851bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:58:34.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f851bfff700 10 monclient: dump: 2023-09-30T01:58:34.570 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:34.571 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:34.571 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:34.571 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:34.571 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:34.571 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:34.571 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:34.571 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:34.571 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:34.571 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:34.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f85227fc700 4 set_mon_vals no callback set 2023-09-30T01:58:34.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f8529f49700 10 monclient: _renew_subs 2023-09-30T01:58:34.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f8529f49700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:34.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.534+0000 7f8529f49700 1 -- 172.21.15.7:0/62759190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8524142460 con 0x7f85240a29e0 2023-09-30T01:58:34.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.535+0000 7f851bfff700 1 -- 172.21.15.7:0/62759190 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f850c014640 con 0x7f85240a29e0 2023-09-30T01:58:34.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.535+0000 7f8529f49700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:34.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.535+0000 7f8529f49700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:34.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.535+0000 7f8529f49700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:34.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.535+0000 7f8529f49700 1 -- 172.21.15.7:0/62759190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85240974f0 con 0x7f85240a29e0 2023-09-30T01:58:34.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.535+0000 7f851bfff700 1 --2- 172.21.15.7:0/62759190 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f85040357e0 0x7f8504037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:34.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.535+0000 7f851bfff700 1 -- 172.21.15.7:0/62759190 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f850c048330 con 0x7f85240a29e0 2023-09-30T01:58:34.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.536+0000 7f85237fe700 1 --2- 172.21.15.7:0/62759190 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f85040357e0 0x7f8504037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:34.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.536+0000 7f85237fe700 10 monclient: get_auth_request con 0x7f85040357e0 auth_method 0 2023-09-30T01:58:34.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.538+0000 7f85237fe700 1 --2- 172.21.15.7:0/62759190 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f85040357e0 0x7f8504037c90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8514009990 tx=0x7f8514016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:34.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.540+0000 7f851bfff700 1 -- 172.21.15.7:0/62759190 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f850c012e40 con 0x7f85240a29e0 2023-09-30T01:58:34.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.540+0000 7f851bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:34.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.540+0000 7f851bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:58:34.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.773+0000 7f8529f49700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:58:34.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.773+0000 7f8529f49700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:34.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.773+0000 7f8529f49700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:34.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.773+0000 7f8529f49700 1 -- 172.21.15.7:0/62759190 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8524064c10 con 0x7f85240a29e0 2023-09-30T01:58:34.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.773+0000 7f851bfff700 1 -- 172.21.15.7:0/62759190 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f850c011340 con 0x7f85240a29e0 2023-09-30T01:58:34.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.773+0000 7f851bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:34.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.773+0000 7f851bfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:58:34.774 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:58:34.774 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:34.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.775+0000 7f8529f49700 1 -- 172.21.15.7:0/62759190 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f85040357e0 msgr2=0x7f8504037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:34.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.775+0000 7f8529f49700 1 --2- 172.21.15.7:0/62759190 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f85040357e0 0x7f8504037c90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8514009990 tx=0x7f8514016000 comp rx=0 tx=0).stop 2023-09-30T01:58:34.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.775+0000 7f8529f49700 10 monclient: shutdown 2023-09-30T01:58:34.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.775+0000 7f8529f49700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:34.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.775+0000 7f8529f49700 1 -- 172.21.15.7:0/62759190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85240a29e0 msgr2=0x7f8524141f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:34.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.775+0000 7f8529f49700 1 --2- 172.21.15.7:0/62759190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85240a29e0 0x7f8524141f20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f850c00d890 tx=0x7f850c008040 comp rx=0 tx=0).stop 2023-09-30T01:58:34.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.775+0000 7f8529f49700 1 -- 172.21.15.7:0/62759190 shutdown_connections 2023-09-30T01:58:34.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.775+0000 7f8529f49700 1 --2- 172.21.15.7:0/62759190 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f85040357e0 0x7f8504037c90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:34.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.775+0000 7f8529f49700 1 --2- 172.21.15.7:0/62759190 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8524132220 0x7f852411cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:34.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.775+0000 7f8529f49700 1 --2- 172.21.15.7:0/62759190 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8524096f90 0x7f85241419e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:34.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.775+0000 7f8529f49700 1 --2- 172.21.15.7:0/62759190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f85240a29e0 0x7f8524141f20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:34.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.775+0000 7f8529f49700 1 -- 172.21.15.7:0/62759190 >> 172.21.15.7:0/62759190 conn(0x7f85240710d0 msgr2=0x7f8524130fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:34.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.775+0000 7f8529f49700 1 -- 172.21.15.7:0/62759190 shutdown_connections 2023-09-30T01:58:34.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:34.775+0000 7f8529f49700 1 -- 172.21.15.7:0/62759190 wait complete. 2023-09-30T01:58:37.788 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:37.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.918+0000 7f428c705700 10 monclient: get_monmap_and_config 2023-09-30T01:58:37.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.918+0000 7f428c705700 10 monclient: build_initial_monmap 2023-09-30T01:58:37.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.918+0000 7f428c705700 10 monclient: monmap: 2023-09-30T01:58:37.919 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:37.919 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:37.919 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:58:37.919723+0000 2023-09-30T01:58:37.919 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:58:37.919723+0000 2023-09-30T01:58:37.919 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:37.919 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:37.919 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:58:37.920 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:58:37.920 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:58:37.920 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:58:37.920 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:58:37.920 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:37.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.919+0000 7f428c705700 1 Processor -- start 2023-09-30T01:58:37.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.919+0000 7f428c705700 1 -- start start 2023-09-30T01:58:37.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.919+0000 7f428c705700 10 monclient: init 2023-09-30T01:58:37.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.919+0000 7f428c705700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:37.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.919+0000 7f428c705700 10 monclient: _add_conns ranks=[1,0,3,2,4] 2023-09-30T01:58:37.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.919+0000 7f428c705700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4284096a50 addr v1:172.21.15.50:6789/0 2023-09-30T01:58:37.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.919+0000 7f428c705700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4284096e20 addr v1:172.21.15.7:6789/0 2023-09-30T01:58:37.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.919+0000 7f428c705700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4284138700 0x7f428413aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:37.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.919+0000 7f428c705700 10 monclient(hunting): picked mon.noname-a con 0x7f4284138700 addr v2:172.21.15.7:3300/0 2023-09-30T01:58:37.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.919+0000 7f428c705700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4284132e60 con 0x7f4284096e20 2023-09-30T01:58:37.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.919+0000 7f428c705700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4284099480 con 0x7f4284096a50 2023-09-30T01:58:37.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f428c705700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:37.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f428c705700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4284097bf0 con 0x7f4284138700 2023-09-30T01:58:37.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f428c705700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:37.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f428c705700 10 monclient(hunting): authenticate will time out at 1406.985028s 2023-09-30T01:58:37.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f428aca2700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4284138700 0x7f428413aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:37.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f4289ca0700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4284096e20 0x7f4284097240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55464/0 (socket says 172.21.15.7:55464) 2023-09-30T01:58:37.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f4289ca0700 1 -- 172.21.15.7:0/3471680506 learned_addr learned my addr 172.21.15.7:0/3471680506 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:58:37.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f428aca2700 10 monclient(hunting): get_auth_request con 0x7f4284138700 auth_method 0 2023-09-30T01:58:37.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f428aca2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:37.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f428aca2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:37.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f428aca2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:37.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f428aca2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:37.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f428aca2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:37.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f428aca2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:37.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f428aca2700 10 monclient(hunting): handle_auth_done global_id 4617 payload 274 2023-09-30T01:58:37.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f428aca2700 1 -- 172.21.15.7:0/3471680506 >> v1:172.21.15.50:6789/0 conn(0x7f4284096a50 legacy=0x7f4284132750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:37.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f428aca2700 1 -- 172.21.15.7:0/3471680506 >> v1:172.21.15.7:6789/0 conn(0x7f4284096e20 legacy=0x7f4284097240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:37.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f428aca2700 10 monclient: _finish_hunting 0 2023-09-30T01:58:37.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f428aca2700 1 monclient: found mon.noname-a 2023-09-30T01:58:37.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f428aca2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:37.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f428aca2700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:58:37.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.920+0000 7f428aca2700 1 -- 172.21.15.7:0/3471680506 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42840991a0 con 0x7f4284138700 2023-09-30T01:58:37.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.922+0000 7f428aca2700 1 -- 172.21.15.7:0/3471680506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284138700 msgr2=0x7f428413aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:37.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.922+0000 7f428aca2700 1 --2- 172.21.15.7:0/3471680506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284138700 0x7f428413aae0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f427c009e60 tx=0x7f427c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ce014d0c3783cc server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:37.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.922+0000 7f428949f700 1 -- 172.21.15.7:0/3471680506 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c00e070 con 0x7f4284138700 2023-09-30T01:58:37.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.922+0000 7f428949f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:37.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.922+0000 7f428949f700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:58:37.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.922+0000 7f428949f700 10 monclient: dump: 2023-09-30T01:58:37.926 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:37.926 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:37.926 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:37.926 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:37.926 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:37.927 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:37.927 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:37.927 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:37.927 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:37.927 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:37.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.922+0000 7f428949f700 10 monclient: _finish_auth 0 2023-09-30T01:58:37.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.922+0000 7f428949f700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:37.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.922+0000 7f428c705700 5 monclient: authenticate success, global_id 4617 2023-09-30T01:58:37.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.922+0000 7f428949f700 1 -- 172.21.15.7:0/3471680506 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f427c012430 con 0x7f4284138700 2023-09-30T01:58:37.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.922+0000 7f428c705700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:58:37.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.922+0000 7f428949f700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:37.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.922+0000 7f428949f700 1 -- 172.21.15.7:0/3471680506 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c00bea0 con 0x7f4284138700 2023-09-30T01:58:37.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.922+0000 7f428c705700 10 monclient: get_monmap_and_config success 2023-09-30T01:58:37.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.922+0000 7f428c705700 4 set_mon_vals no callback set 2023-09-30T01:58:37.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.923+0000 7f428949f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:37.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.923+0000 7f428949f700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:58:37.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.923+0000 7f428949f700 10 monclient: dump: 2023-09-30T01:58:37.928 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:37.928 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:37.928 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:37.928 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:37.928 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:37.928 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:37.929 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:37.929 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:37.929 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:37.929 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:37.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.923+0000 7f428c705700 10 monclient: shutdown 2023-09-30T01:58:37.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.923+0000 7f428c705700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:37.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.923+0000 7f428c705700 1 -- 172.21.15.7:0/3471680506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284138700 msgr2=0x7f428413aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:37.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.923+0000 7f428c705700 1 --2- 172.21.15.7:0/3471680506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284138700 0x7f428413aae0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f427c009e60 tx=0x7f427c009780 comp rx=0 tx=0).stop 2023-09-30T01:58:37.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.923+0000 7f428c705700 1 -- 172.21.15.7:0/3471680506 shutdown_connections 2023-09-30T01:58:37.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.923+0000 7f428c705700 1 --2- 172.21.15.7:0/3471680506 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284138700 0x7f428413aae0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:37.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.923+0000 7f428c705700 1 -- 172.21.15.7:0/3471680506 >> 172.21.15.7:0/3471680506 conn(0x7f4284070cf0 msgr2=0x7f42840710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:37.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.923+0000 7f428c705700 1 -- 172.21.15.7:0/3471680506 shutdown_connections 2023-09-30T01:58:37.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.923+0000 7f428c705700 1 -- 172.21.15.7:0/3471680506 wait complete. 2023-09-30T01:58:37.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.924+0000 7f428c705700 10 monclient: build_initial_monmap 2023-09-30T01:58:37.938 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.924+0000 7f428c705700 10 monclient: monmap: 2023-09-30T01:58:37.939 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:37.939 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:58:37.939 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:58:37.939 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:58:37.939 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:37.939 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:37.939 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:58:37.939 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:58:37.939 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:58:37.939 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:37.939 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.924+0000 7f428c705700 1 Processor -- start 2023-09-30T01:58:37.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.924+0000 7f428c705700 1 -- start start 2023-09-30T01:58:37.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.924+0000 7f428c705700 10 monclient: init 2023-09-30T01:58:37.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.924+0000 7f428c705700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:37.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.924+0000 7f428c705700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:58:37.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.924+0000 7f428c705700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4284096a50 0x7f42840a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:37.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.924+0000 7f428c705700 10 monclient(hunting): picked mon.noname-b con 0x7f4284096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:58:37.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.924+0000 7f428c705700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4284096e20 0x7f428409c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:37.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.924+0000 7f428c705700 10 monclient(hunting): picked mon.noname-c con 0x7f4284096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:37.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.924+0000 7f428c705700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284138700 0x7f428409cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:37.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.924+0000 7f428c705700 10 monclient(hunting): picked mon.noname-a con 0x7f4284138700 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:58:37.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.924+0000 7f428c705700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:37.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.924+0000 7f428c705700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42840a6420 con 0x7f4284138700 2023-09-30T01:58:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.924+0000 7f428c705700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.924+0000 7f428c705700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428409fdb0 con 0x7f4284096a50 2023-09-30T01:58:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.924+0000 7f428c705700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.924+0000 7f428c705700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42840a0090 con 0x7f4284096e20 2023-09-30T01:58:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.924+0000 7f428c705700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.924+0000 7f428c705700 10 monclient(hunting): authenticate will time out at 1406.989979s 2023-09-30T01:58:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428aca2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284138700 0x7f428409cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f4289ca0700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4284096e20 0x7f428409c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f4289ca0700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4284096e20 0x7f428409c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:51274/0 (socket says 172.21.15.7:51274) 2023-09-30T01:58:37.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428aca2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284138700 0x7f428409cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44824/0 (socket says 172.21.15.7:44824) 2023-09-30T01:58:37.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f4289ca0700 1 -- 172.21.15.7:0/3829253991 learned_addr learned my addr 172.21.15.7:0/3829253991 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:37.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f4289ca0700 10 monclient(hunting): get_auth_request con 0x7f4284096e20 auth_method 0 2023-09-30T01:58:37.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f4289ca0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:37.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f4289ca0700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:37.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f4289ca0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:37.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428aca2700 10 monclient(hunting): get_auth_request con 0x7f4284138700 auth_method 0 2023-09-30T01:58:37.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428aca2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:37.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428aca2700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:37.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428aca2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:37.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428a4a1700 1 --2- 172.21.15.7:0/3829253991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4284096a50 0x7f42840a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:37.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428a4a1700 10 monclient(hunting): get_auth_request con 0x7f4284096a50 auth_method 0 2023-09-30T01:58:37.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428a4a1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:37.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428a4a1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:37.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428a4a1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:37.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428aca2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:37.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428aca2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:37.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428aca2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:37.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f4289ca0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:37.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f4289ca0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:37.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f4289ca0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:37.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428a4a1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:37.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428a4a1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:37.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428a4a1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:37.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428aca2700 10 monclient(hunting): handle_auth_done global_id 4620 payload 824 2023-09-30T01:58:37.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428aca2700 1 -- 172.21.15.7:0/3829253991 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4284096e20 msgr2=0x7f428409c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:37.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428aca2700 1 --2- 172.21.15.7:0/3829253991 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4284096e20 0x7f428409c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:37.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428aca2700 1 -- 172.21.15.7:0/3829253991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4284096a50 msgr2=0x7f42840a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:37.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428aca2700 1 --2- 172.21.15.7:0/3829253991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4284096a50 0x7f42840a0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:37.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428aca2700 10 monclient: _finish_hunting 0 2023-09-30T01:58:37.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428aca2700 1 monclient: found mon.noname-a 2023-09-30T01:58:37.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428aca2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:37.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428aca2700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:58:37.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f428aca2700 1 -- 172.21.15.7:0/3829253991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427c009a20 con 0x7f4284138700 2023-09-30T01:58:37.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.925+0000 7f4289ca0700 1 --2- 172.21.15.7:0/3829253991 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4284096e20 0x7f428409c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:58:37.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.927+0000 7f428aca2700 1 -- 172.21.15.7:0/3829253991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284138700 msgr2=0x7f428409cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:37.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.927+0000 7f428aca2700 1 --2- 172.21.15.7:0/3829253991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284138700 0x7f428409cbe0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f42841331d0 tx=0x7f427c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:37.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.927+0000 7f427a7fc700 1 -- 172.21.15.7:0/3829253991 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c026430 con 0x7f4284138700 2023-09-30T01:58:37.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.927+0000 7f427a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:37.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.927+0000 7f427a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:58:37.945 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.927+0000 7f427a7fc700 10 monclient: dump: 2023-09-30T01:58:37.945 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:37.945 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:37.946 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:37.946 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:37.946 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:37.946 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:37.946 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:37.946 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:37.946 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:37.946 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:37.946 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.927+0000 7f427a7fc700 10 monclient: _finish_auth 0 2023-09-30T01:58:37.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.927+0000 7f427a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:37.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.927+0000 7f428c705700 5 monclient: authenticate success, global_id 4620 2023-09-30T01:58:37.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.927+0000 7f427a7fc700 1 -- 172.21.15.7:0/3829253991 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f427c021070 con 0x7f4284138700 2023-09-30T01:58:37.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.927+0000 7f428c705700 10 monclient: _renew_subs 2023-09-30T01:58:37.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.927+0000 7f428c705700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:37.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.927+0000 7f428c705700 1 -- 172.21.15.7:0/3829253991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f428409e280 con 0x7f4284138700 2023-09-30T01:58:37.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.927+0000 7f427a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:37.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.927+0000 7f427a7fc700 1 -- 172.21.15.7:0/3829253991 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c0267e0 con 0x7f4284138700 2023-09-30T01:58:37.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.927+0000 7f427a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:37.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.927+0000 7f427a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:58:37.948 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.927+0000 7f427a7fc700 10 monclient: dump: 2023-09-30T01:58:37.948 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:37.948 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:37.948 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:37.948 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:37.948 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:37.948 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:37.948 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:37.948 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:37.949 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:37.949 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:37.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.927+0000 7f4288c9e700 4 set_mon_vals no callback set 2023-09-30T01:58:37.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.928+0000 7f428c705700 10 monclient: _renew_subs 2023-09-30T01:58:37.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.928+0000 7f428c705700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:37.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.928+0000 7f428c705700 1 -- 172.21.15.7:0/3829253991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42840a0fb0 con 0x7f4284138700 2023-09-30T01:58:37.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.928+0000 7f427a7fc700 1 -- 172.21.15.7:0/3829253991 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f427c023030 con 0x7f4284138700 2023-09-30T01:58:37.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.928+0000 7f428c705700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:37.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.928+0000 7f428c705700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:37.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.928+0000 7f428c705700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:37.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.928+0000 7f428c705700 1 -- 172.21.15.7:0/3829253991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42840711e0 con 0x7f4284138700 2023-09-30T01:58:37.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.929+0000 7f427a7fc700 1 --2- 172.21.15.7:0/3829253991 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f42680357e0 0x7f4268037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:37.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.929+0000 7f427a7fc700 1 -- 172.21.15.7:0/3829253991 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f427c049f60 con 0x7f4284138700 2023-09-30T01:58:37.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.929+0000 7f428a4a1700 1 --2- 172.21.15.7:0/3829253991 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f42680357e0 0x7f4268037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:37.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.929+0000 7f428a4a1700 10 monclient: get_auth_request con 0x7f42680357e0 auth_method 0 2023-09-30T01:58:37.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.931+0000 7f428a4a1700 1 --2- 172.21.15.7:0/3829253991 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f42680357e0 0x7f4268037c90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4280009af0 tx=0x7f4280009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:37.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.933+0000 7f427a7fc700 1 -- 172.21.15.7:0/3829253991 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f427c01b7b0 con 0x7f4284138700 2023-09-30T01:58:37.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.933+0000 7f427a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:37.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:37.933+0000 7f427a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:58:38.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:38.172+0000 7f428c705700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:58:38.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:38.172+0000 7f428c705700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:38.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:38.172+0000 7f428c705700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:38.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:38.172+0000 7f428c705700 1 -- 172.21.15.7:0/3829253991 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f42840711e0 con 0x7f4284138700 2023-09-30T01:58:38.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:38.173+0000 7f427a7fc700 1 -- 172.21.15.7:0/3829253991 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f427c01a3c0 con 0x7f4284138700 2023-09-30T01:58:38.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:38.173+0000 7f427a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:38.173 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:38.173+0000 7f427a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:58:38.174 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:58:38.174 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:38.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:38.174+0000 7f428c705700 1 -- 172.21.15.7:0/3829253991 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f42680357e0 msgr2=0x7f4268037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:38.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:38.175+0000 7f428c705700 1 --2- 172.21.15.7:0/3829253991 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f42680357e0 0x7f4268037c90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4280009af0 tx=0x7f4280009060 comp rx=0 tx=0).stop 2023-09-30T01:58:38.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:38.175+0000 7f428c705700 10 monclient: shutdown 2023-09-30T01:58:38.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:38.175+0000 7f428c705700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:38.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:38.175+0000 7f428c705700 1 -- 172.21.15.7:0/3829253991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284138700 msgr2=0x7f428409cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:38.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:38.175+0000 7f428c705700 1 --2- 172.21.15.7:0/3829253991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284138700 0x7f428409cbe0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f42841331d0 tx=0x7f427c018000 comp rx=0 tx=0).stop 2023-09-30T01:58:38.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:38.175+0000 7f428c705700 1 -- 172.21.15.7:0/3829253991 shutdown_connections 2023-09-30T01:58:38.175 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:38.175+0000 7f428c705700 1 --2- 172.21.15.7:0/3829253991 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f42680357e0 0x7f4268037c90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:38.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:38.175+0000 7f428c705700 1 --2- 172.21.15.7:0/3829253991 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4284138700 0x7f428409cbe0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:38.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:38.175+0000 7f428c705700 1 --2- 172.21.15.7:0/3829253991 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4284096a50 0x7f42840a0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:38.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:38.175+0000 7f428c705700 1 --2- 172.21.15.7:0/3829253991 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f4284096e20 0x7f428409c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:38.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:38.175+0000 7f428c705700 1 -- 172.21.15.7:0/3829253991 >> 172.21.15.7:0/3829253991 conn(0x7f4284070cf0 msgr2=0x7f4284139bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:38.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:38.175+0000 7f428c705700 1 -- 172.21.15.7:0/3829253991 shutdown_connections 2023-09-30T01:58:38.176 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:38.175+0000 7f428c705700 1 -- 172.21.15.7:0/3829253991 wait complete. 2023-09-30T01:58:41.187 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:41.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.316+0000 7fb2ba734700 10 monclient: get_monmap_and_config 2023-09-30T01:58:41.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.316+0000 7fb2ba734700 10 monclient: build_initial_monmap 2023-09-30T01:58:41.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.316+0000 7fb2ba734700 10 monclient: monmap: 2023-09-30T01:58:41.316 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:41.316 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:41.317 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:58:41.317434+0000 2023-09-30T01:58:41.317 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:58:41.317434+0000 2023-09-30T01:58:41.317 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:41.317 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:41.317 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:58:41.317 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:58:41.317 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:58:41.317 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:58:41.317 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:58:41.317 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:41.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2ba734700 1 Processor -- start 2023-09-30T01:58:41.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2ba734700 1 -- start start 2023-09-30T01:58:41.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2ba734700 10 monclient: init 2023-09-30T01:58:41.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2ba734700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:41.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2ba734700 10 monclient: _add_conns ranks=[3,1,2,0,4] 2023-09-30T01:58:41.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2ba734700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb2b4096a50 0x7fb2b41327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:41.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2ba734700 10 monclient(hunting): picked mon.noname-a con 0x7fb2b4096a50 addr v2:172.21.15.7:3300/0 2023-09-30T01:58:41.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2ba734700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb2b40a2390 addr v1:172.21.15.50:6789/0 2023-09-30T01:58:41.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2ba734700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb2b4097530 0x7fb2b4139720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:41.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2ba734700 10 monclient(hunting): picked mon.noname-b con 0x7fb2b4097530 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:41.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2ba734700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2b40980e0 con 0x7fb2b40a2390 2023-09-30T01:58:41.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2ba734700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:41.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2ba734700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2b40991a0 con 0x7fb2b4096a50 2023-09-30T01:58:41.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2ba734700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:41.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2ba734700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b4099480 con 0x7fb2b4097530 2023-09-30T01:58:41.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2ba734700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:41.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2ba734700 10 monclient(hunting): authenticate will time out at 1410.382769s 2023-09-30T01:58:41.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2b3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb2b4096a50 0x7fb2b41327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:41.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2b8cd1700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb2b4097530 0x7fb2b4139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:41.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2b3fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb2b4096a50 0x7fb2b41327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44834/0 (socket says 172.21.15.7:44834) 2023-09-30T01:58:41.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2b8cd1700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb2b4097530 0x7fb2b4139720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:51290/0 (socket says 172.21.15.7:51290) 2023-09-30T01:58:41.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2b3fff700 1 -- 172.21.15.7:0/3849708273 learned_addr learned my addr 172.21.15.7:0/3849708273 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2b3fff700 10 monclient(hunting): get_auth_request con 0x7fb2b4096a50 auth_method 0 2023-09-30T01:58:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2b3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2b3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2b3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2b8cd1700 10 monclient(hunting): get_auth_request con 0x7fb2b4097530 auth_method 0 2023-09-30T01:58:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2b8cd1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2b8cd1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.317+0000 7fb2b8cd1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.318+0000 7fb2b3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.318+0000 7fb2b3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:41.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.318+0000 7fb2b3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:41.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.318+0000 7fb2b8cd1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:41.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.318+0000 7fb2b8cd1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:41.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.318+0000 7fb2b8cd1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:41.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.318+0000 7fb2b3fff700 10 monclient(hunting): handle_auth_done global_id 4623 payload 274 2023-09-30T01:58:41.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.318+0000 7fb2b3fff700 1 -- 172.21.15.7:0/3849708273 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb2b4097530 msgr2=0x7fb2b4139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:41.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.318+0000 7fb2b3fff700 1 --2- 172.21.15.7:0/3849708273 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb2b4097530 0x7fb2b4139720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:41.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.318+0000 7fb2b3fff700 1 -- 172.21.15.7:0/3849708273 >> v1:172.21.15.50:6789/0 conn(0x7fb2b40a2390 legacy=0x7fb2b4096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:41.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.318+0000 7fb2b3fff700 10 monclient: _finish_hunting 0 2023-09-30T01:58:41.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.318+0000 7fb2b3fff700 1 monclient: found mon.noname-a 2023-09-30T01:58:41.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.318+0000 7fb2b3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:41.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.318+0000 7fb2b3fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:58:41.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.318+0000 7fb2b3fff700 1 -- 172.21.15.7:0/3849708273 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b4097bf0 con 0x7fb2b4096a50 2023-09-30T01:58:41.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.318+0000 7fb2b8cd1700 1 --2- 172.21.15.7:0/3849708273 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb2b4097530 0x7fb2b4139720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:58:41.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.320+0000 7fb2b3fff700 1 -- 172.21.15.7:0/3849708273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2b4096a50 msgr2=0x7fb2b41327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:41.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.320+0000 7fb2b3fff700 1 --2- 172.21.15.7:0/3849708273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2b4096a50 0x7fb2b41327a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb2a40099b0 tx=0x7fb2a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79c4586eac04223c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:41.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.320+0000 7fb2b2ffd700 1 -- 172.21.15.7:0/3849708273 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a4009ea0 con 0x7fb2b4096a50 2023-09-30T01:58:41.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.320+0000 7fb2b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:41.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.320+0000 7fb2b2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:58:41.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.320+0000 7fb2b2ffd700 10 monclient: dump: 2023-09-30T01:58:41.326 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:41.326 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:41.327 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:41.327 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:41.327 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:41.327 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:41.327 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:41.327 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:41.327 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:41.327 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:41.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.320+0000 7fb2b2ffd700 10 monclient: _finish_auth 0 2023-09-30T01:58:41.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.320+0000 7fb2b2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:41.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.320+0000 7fb2ba734700 5 monclient: authenticate success, global_id 4623 2023-09-30T01:58:41.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.320+0000 7fb2b2ffd700 1 -- 172.21.15.7:0/3849708273 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2a400ba30 con 0x7fb2b4096a50 2023-09-30T01:58:41.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.320+0000 7fb2ba734700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:58:41.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.320+0000 7fb2b2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:41.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.320+0000 7fb2ba734700 10 monclient: get_monmap_and_config success 2023-09-30T01:58:41.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.320+0000 7fb2ba734700 4 set_mon_vals no callback set 2023-09-30T01:58:41.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.320+0000 7fb2b2ffd700 1 -- 172.21.15.7:0/3849708273 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a400bdf0 con 0x7fb2b4096a50 2023-09-30T01:58:41.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.320+0000 7fb2b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:41.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.320+0000 7fb2b2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:58:41.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.320+0000 7fb2b2ffd700 10 monclient: dump: 2023-09-30T01:58:41.329 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:41.329 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:41.329 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:41.329 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:41.329 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:41.329 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:41.329 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:41.329 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:41.329 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:41.329 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:41.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.321+0000 7fb2ba734700 10 monclient: shutdown 2023-09-30T01:58:41.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.321+0000 7fb2ba734700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:41.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.321+0000 7fb2ba734700 1 -- 172.21.15.7:0/3849708273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2b4096a50 msgr2=0x7fb2b41327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:41.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.321+0000 7fb2ba734700 1 --2- 172.21.15.7:0/3849708273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2b4096a50 0x7fb2b41327a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb2a40099b0 tx=0x7fb2a4009250 comp rx=0 tx=0).stop 2023-09-30T01:58:41.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.321+0000 7fb2ba734700 1 -- 172.21.15.7:0/3849708273 shutdown_connections 2023-09-30T01:58:41.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.321+0000 7fb2ba734700 1 --2- 172.21.15.7:0/3849708273 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb2b4097530 0x7fb2b4139720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:41.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.321+0000 7fb2ba734700 1 --2- 172.21.15.7:0/3849708273 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2b4096a50 0x7fb2b41327a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:41.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.321+0000 7fb2ba734700 1 -- 172.21.15.7:0/3849708273 >> 172.21.15.7:0/3849708273 conn(0x7fb2b4070cf0 msgr2=0x7fb2b40710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:41.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.321+0000 7fb2ba734700 1 -- 172.21.15.7:0/3849708273 shutdown_connections 2023-09-30T01:58:41.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.321+0000 7fb2ba734700 1 -- 172.21.15.7:0/3849708273 wait complete. 2023-09-30T01:58:41.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.321+0000 7fb2ba734700 10 monclient: build_initial_monmap 2023-09-30T01:58:41.330 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.321+0000 7fb2ba734700 10 monclient: monmap: 2023-09-30T01:58:41.331 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:41.331 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:58:41.331 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:58:41.331 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:58:41.331 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:41.331 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:41.331 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:58:41.331 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:58:41.331 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:58:41.331 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:41.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.322+0000 7fb2ba734700 1 Processor -- start 2023-09-30T01:58:41.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.322+0000 7fb2ba734700 1 -- start start 2023-09-30T01:58:41.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.322+0000 7fb2ba734700 10 monclient: init 2023-09-30T01:58:41.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.322+0000 7fb2ba734700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:41.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.322+0000 7fb2ba734700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:58:41.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.322+0000 7fb2ba734700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2b4096a50 0x7fb2b40a0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:41.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.322+0000 7fb2ba734700 10 monclient(hunting): picked mon.noname-a con 0x7fb2b4096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:58:41.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.322+0000 7fb2ba734700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb2b4097530 0x7fb2b409c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:41.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.322+0000 7fb2ba734700 10 monclient(hunting): picked mon.noname-c con 0x7fb2b4097530 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:41.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.322+0000 7fb2ba734700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2b40a2390 0x7fb2b409cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:41.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.322+0000 7fb2ba734700 10 monclient(hunting): picked mon.noname-b con 0x7fb2b40a2390 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:58:41.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.322+0000 7fb2ba734700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:41.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.322+0000 7fb2ba734700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b41334e0 con 0x7fb2b4096a50 2023-09-30T01:58:41.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.322+0000 7fb2ba734700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:41.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.322+0000 7fb2ba734700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b4132320 con 0x7fb2b40a2390 2023-09-30T01:58:41.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.322+0000 7fb2ba734700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:41.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.322+0000 7fb2ba734700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b409e7f0 con 0x7fb2b4097530 2023-09-30T01:58:41.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.322+0000 7fb2ba734700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:41.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.322+0000 7fb2ba734700 10 monclient(hunting): authenticate will time out at 1410.387948s 2023-09-30T01:58:41.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2b4096a50 0x7fb2b40a0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:41.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b37fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb2b4097530 0x7fb2b409c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:41.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b3fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2b4096a50 0x7fb2b40a0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44848/0 (socket says 172.21.15.7:44848) 2023-09-30T01:58:41.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b3fff700 1 -- 172.21.15.7:0/3672699485 learned_addr learned my addr 172.21.15.7:0/3672699485 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:41.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b3fff700 10 monclient(hunting): get_auth_request con 0x7fb2b4096a50 auth_method 0 2023-09-30T01:58:41.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:41.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:41.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:41.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b37fe700 10 monclient(hunting): get_auth_request con 0x7fb2b4097530 auth_method 0 2023-09-30T01:58:41.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:41.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:41.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:41.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:41.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:41.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b8cd1700 1 --2- 172.21.15.7:0/3672699485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2b40a2390 0x7fb2b409cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:41.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:41.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:41.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:41.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:41.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b8cd1700 10 monclient(hunting): get_auth_request con 0x7fb2b40a2390 auth_method 0 2023-09-30T01:58:41.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b8cd1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:41.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b8cd1700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:41.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b8cd1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:41.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b8cd1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:41.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b8cd1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:41.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b8cd1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:41.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b3fff700 10 monclient(hunting): handle_auth_done global_id 4626 payload 824 2023-09-30T01:58:41.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b3fff700 1 -- 172.21.15.7:0/3672699485 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb2b4097530 msgr2=0x7fb2b409c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:41.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b3fff700 1 --2- 172.21.15.7:0/3672699485 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb2b4097530 0x7fb2b409c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:41.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b3fff700 1 -- 172.21.15.7:0/3672699485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2b40a2390 msgr2=0x7fb2b409cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:41.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b3fff700 1 --2- 172.21.15.7:0/3672699485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2b40a2390 0x7fb2b409cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:41.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b3fff700 10 monclient: _finish_hunting 0 2023-09-30T01:58:41.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b3fff700 1 monclient: found mon.noname-a 2023-09-30T01:58:41.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:41.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b3fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:58:41.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b3fff700 1 -- 172.21.15.7:0/3672699485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ac009a20 con 0x7fb2b4096a50 2023-09-30T01:58:41.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.323+0000 7fb2b37fe700 1 --2- 172.21.15.7:0/3672699485 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb2b4097530 0x7fb2b409c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:58:41.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb2b3fff700 1 -- 172.21.15.7:0/3672699485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2b4096a50 msgr2=0x7fb2b40a0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:41.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb2b3fff700 1 --2- 172.21.15.7:0/3672699485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2b4096a50 0x7fb2b40a0a60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb2b40a2b30 tx=0x7fb2a4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:41.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb29bfff700 1 -- 172.21.15.7:0/3672699485 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a4006b80 con 0x7fb2b4096a50 2023-09-30T01:58:41.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb29bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:41.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb29bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:58:41.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb29bfff700 10 monclient: dump: 2023-09-30T01:58:41.344 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:41.344 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:41.344 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:41.344 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:41.344 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:41.344 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:41.344 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:41.344 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:41.344 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:41.345 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:41.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb29bfff700 10 monclient: _finish_auth 0 2023-09-30T01:58:41.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb29bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:41.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb29bfff700 1 -- 172.21.15.7:0/3672699485 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2a4020070 con 0x7fb2b4096a50 2023-09-30T01:58:41.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb2ba734700 5 monclient: authenticate success, global_id 4626 2023-09-30T01:58:41.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb2ba734700 10 monclient: _renew_subs 2023-09-30T01:58:41.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb2ba734700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:41.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb2ba734700 1 -- 172.21.15.7:0/3672699485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a40094f0 con 0x7fb2b4096a50 2023-09-30T01:58:41.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb29bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:41.345 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb29bfff700 1 -- 172.21.15.7:0/3672699485 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a4011e00 con 0x7fb2b4096a50 2023-09-30T01:58:41.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb29bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:41.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb29bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:58:41.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb29bfff700 10 monclient: dump: 2023-09-30T01:58:41.346 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:41.346 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:41.346 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:41.346 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:41.346 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:41.346 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:41.346 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:41.347 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:41.347 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:41.347 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:41.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb2ba734700 10 monclient: _renew_subs 2023-09-30T01:58:41.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb2ba734700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:41.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb2b27fc700 4 set_mon_vals no callback set 2023-09-30T01:58:41.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.325+0000 7fb2ba734700 1 -- 172.21.15.7:0/3672699485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2b40a0fa0 con 0x7fb2b4096a50 2023-09-30T01:58:41.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.326+0000 7fb29bfff700 1 -- 172.21.15.7:0/3672699485 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fb2a4022030 con 0x7fb2b4096a50 2023-09-30T01:58:41.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.326+0000 7fb2ba734700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:41.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.326+0000 7fb2ba734700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:41.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.326+0000 7fb2ba734700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:41.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.326+0000 7fb2ba734700 1 -- 172.21.15.7:0/3672699485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2b40a1280 con 0x7fb2b4096a50 2023-09-30T01:58:41.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.326+0000 7fb29bfff700 1 --2- 172.21.15.7:0/3672699485 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb2a00357e0 0x7fb2a0037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:41.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.326+0000 7fb29bfff700 1 -- 172.21.15.7:0/3672699485 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fb2a4049940 con 0x7fb2b4096a50 2023-09-30T01:58:41.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.327+0000 7fb2b37fe700 1 --2- 172.21.15.7:0/3672699485 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb2a00357e0 0x7fb2a0037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:41.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.327+0000 7fb2b37fe700 10 monclient: get_auth_request con 0x7fb2a00357e0 auth_method 0 2023-09-30T01:58:41.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.329+0000 7fb2b37fe700 1 --2- 172.21.15.7:0/3672699485 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb2a00357e0 0x7fb2a0037c90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8002e00 tx=0x7fb2a8007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:41.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.331+0000 7fb29bfff700 1 -- 172.21.15.7:0/3672699485 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb2b40a1280 con 0x7fb2b4096a50 2023-09-30T01:58:41.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.331+0000 7fb29bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:41.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.331+0000 7fb29bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:58:41.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.568+0000 7fb2ba734700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:58:41.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.568+0000 7fb2ba734700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:41.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.568+0000 7fb2ba734700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:41.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.568+0000 7fb2ba734700 1 -- 172.21.15.7:0/3672699485 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb2b40a1280 con 0x7fb2b4096a50 2023-09-30T01:58:41.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.568+0000 7fb29bfff700 1 -- 172.21.15.7:0/3672699485 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb2b40a1280 con 0x7fb2b4096a50 2023-09-30T01:58:41.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.568+0000 7fb29bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:41.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.568+0000 7fb29bfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:58:41.569 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:58:41.569 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.570+0000 7fb2ba734700 1 -- 172.21.15.7:0/3672699485 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb2a00357e0 msgr2=0x7fb2a0037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.570+0000 7fb2ba734700 1 --2- 172.21.15.7:0/3672699485 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb2a00357e0 0x7fb2a0037c90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8002e00 tx=0x7fb2a8007ea0 comp rx=0 tx=0).stop 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.570+0000 7fb2ba734700 10 monclient: shutdown 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.570+0000 7fb2ba734700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.570+0000 7fb2ba734700 1 -- 172.21.15.7:0/3672699485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2b4096a50 msgr2=0x7fb2b40a0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.570+0000 7fb2ba734700 1 --2- 172.21.15.7:0/3672699485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2b4096a50 0x7fb2b40a0a60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb2b40a2b30 tx=0x7fb2a4016000 comp rx=0 tx=0).stop 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.570+0000 7fb2ba734700 1 -- 172.21.15.7:0/3672699485 shutdown_connections 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.570+0000 7fb2ba734700 1 --2- 172.21.15.7:0/3672699485 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb2a00357e0 0x7fb2a0037c90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.570+0000 7fb2ba734700 1 --2- 172.21.15.7:0/3672699485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2b4096a50 0x7fb2b40a0a60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.570+0000 7fb2ba734700 1 --2- 172.21.15.7:0/3672699485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2b40a2390 0x7fb2b409cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:41.571 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.570+0000 7fb2ba734700 1 --2- 172.21.15.7:0/3672699485 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb2b4097530 0x7fb2b409c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:41.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.570+0000 7fb2ba734700 1 -- 172.21.15.7:0/3672699485 >> 172.21.15.7:0/3672699485 conn(0x7fb2b4070cf0 msgr2=0x7fb2b40a44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:41.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.570+0000 7fb2ba734700 1 -- 172.21.15.7:0/3672699485 shutdown_connections 2023-09-30T01:58:41.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:41.570+0000 7fb2ba734700 1 -- 172.21.15.7:0/3672699485 wait complete. 2023-09-30T01:58:44.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:44.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.714+0000 7f194025c700 10 monclient: get_monmap_and_config 2023-09-30T01:58:44.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.714+0000 7f194025c700 10 monclient: build_initial_monmap 2023-09-30T01:58:44.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.714+0000 7f194025c700 10 monclient: monmap: 2023-09-30T01:58:44.715 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:44.715 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:44.715 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:58:44.715190+0000 2023-09-30T01:58:44.715 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:58:44.715190+0000 2023-09-30T01:58:44.715 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:44.715 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:44.715 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:58:44.715 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:58:44.715 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:58:44.715 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:58:44.715 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:58:44.716 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:44.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.714+0000 7f194025c700 1 Processor -- start 2023-09-30T01:58:44.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.714+0000 7f194025c700 1 -- start start 2023-09-30T01:58:44.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.714+0000 7f194025c700 10 monclient: init 2023-09-30T01:58:44.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f194025c700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:44.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f194025c700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2023-09-30T01:58:44.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f194025c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f19380a2170 addr v1:172.21.15.50:6789/0 2023-09-30T01:58:44.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f194025c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f193806cde0 0x7f193806c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:44.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f194025c700 10 monclient(hunting): picked mon.noname-a con 0x7f193806cde0 addr v2:172.21.15.7:3300/0 2023-09-30T01:58:44.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f194025c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f193813bb50 0x7f193813df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:44.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f194025c700 10 monclient(hunting): picked mon.noname-b con 0x7f193813bb50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:44.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f194025c700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19380a2540 con 0x7f19380a2170 2023-09-30T01:58:44.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f194025c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:44.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f194025c700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f193806cba0 con 0x7f193806cde0 2023-09-30T01:58:44.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f194025c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:44.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f194025c700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193806f8b0 con 0x7f193813bb50 2023-09-30T01:58:44.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f194025c700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:44.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f194025c700 10 monclient(hunting): authenticate will time out at 1413.780507s 2023-09-30T01:58:44.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193e7f9700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f193813bb50 0x7f193813df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:44.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193d7f7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f193806cde0 0x7f193806c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:44.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193e7f9700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f193813bb50 0x7f193813df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:50192/0 (socket says 172.21.15.7:50192) 2023-09-30T01:58:44.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193d7f7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f193806cde0 0x7f193806c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37648/0 (socket says 172.21.15.7:37648) 2023-09-30T01:58:44.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193e7f9700 1 -- 172.21.15.7:0/2110336058 learned_addr learned my addr 172.21.15.7:0/2110336058 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:44.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193e7f9700 10 monclient(hunting): get_auth_request con 0x7f193813bb50 auth_method 0 2023-09-30T01:58:44.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193e7f9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:44.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193e7f9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:44.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193e7f9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:44.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193d7f7700 10 monclient(hunting): get_auth_request con 0x7f193806cde0 auth_method 0 2023-09-30T01:58:44.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193d7f7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:44.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193d7f7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:44.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193d7f7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:44.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193e7f9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:44.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193e7f9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:44.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193e7f9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:44.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193d7f7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:44.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193d7f7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:44.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193d7f7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:44.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193e7f9700 10 monclient(hunting): handle_auth_done global_id 4649 payload 274 2023-09-30T01:58:44.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193e7f9700 1 -- 172.21.15.7:0/2110336058 >> v2:172.21.15.7:3300/0 conn(0x7f193806cde0 msgr2=0x7f193806c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:44.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.715+0000 7f193e7f9700 1 --2- 172.21.15.7:0/2110336058 >> v2:172.21.15.7:3300/0 conn(0x7f193806cde0 0x7f193806c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:44.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.716+0000 7f193e7f9700 1 -- 172.21.15.7:0/2110336058 >> v1:172.21.15.50:6789/0 conn(0x7f19380a2170 legacy=0x7f193806b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:44.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.716+0000 7f193e7f9700 10 monclient: _finish_hunting 0 2023-09-30T01:58:44.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.716+0000 7f193e7f9700 1 monclient: found mon.noname-b 2023-09-30T01:58:44.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.716+0000 7f193e7f9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:44.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.716+0000 7f193e7f9700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:58:44.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.716+0000 7f193e7f9700 1 -- 172.21.15.7:0/2110336058 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193806e710 con 0x7f193813bb50 2023-09-30T01:58:44.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.716+0000 7f193d7f7700 1 --2- 172.21.15.7:0/2110336058 >> v2:172.21.15.7:3300/0 conn(0x7f193806cde0 0x7f193806c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:58:44.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.718+0000 7f193e7f9700 1 -- 172.21.15.7:0/2110336058 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f193813bb50 msgr2=0x7f193813df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:44.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.718+0000 7f193e7f9700 1 --2- 172.21.15.7:0/2110336058 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f193813bb50 0x7f193813df30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f19300099b0 tx=0x7f1930009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e11868710e83bcb1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:44.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.718+0000 7f193cff6700 1 -- 172.21.15.7:0/2110336058 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1930009ea0 con 0x7f193813bb50 2023-09-30T01:58:44.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.718+0000 7f193cff6700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:44.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.718+0000 7f193cff6700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:58:44.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.718+0000 7f193cff6700 10 monclient: dump: 2023-09-30T01:58:44.728 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:44.728 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:44.728 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:44.728 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:44.728 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:44.728 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:44.728 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:44.728 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:44.728 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:44.728 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:44.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.718+0000 7f193cff6700 10 monclient: _finish_auth 0 2023-09-30T01:58:44.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.718+0000 7f193cff6700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:44.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.718+0000 7f193cff6700 1 -- 172.21.15.7:0/2110336058 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193000ba30 con 0x7f193813bb50 2023-09-30T01:58:44.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.718+0000 7f194025c700 5 monclient: authenticate success, global_id 4649 2023-09-30T01:58:44.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.718+0000 7f194025c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:58:44.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.718+0000 7f193cff6700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:44.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.718+0000 7f194025c700 10 monclient: get_monmap_and_config success 2023-09-30T01:58:44.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.718+0000 7f193cff6700 1 -- 172.21.15.7:0/2110336058 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193000bdf0 con 0x7f193813bb50 2023-09-30T01:58:44.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.718+0000 7f194025c700 4 set_mon_vals no callback set 2023-09-30T01:58:44.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.718+0000 7f193cff6700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:44.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.718+0000 7f193cff6700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:58:44.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.718+0000 7f193cff6700 10 monclient: dump: 2023-09-30T01:58:44.730 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:44.730 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:44.730 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:44.730 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:44.730 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:44.730 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:44.730 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:44.730 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:44.730 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:44.730 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:44.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.719+0000 7f194025c700 10 monclient: shutdown 2023-09-30T01:58:44.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.719+0000 7f194025c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:44.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.719+0000 7f194025c700 1 -- 172.21.15.7:0/2110336058 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f193813bb50 msgr2=0x7f193813df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:44.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.719+0000 7f194025c700 1 --2- 172.21.15.7:0/2110336058 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f193813bb50 0x7f193813df30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f19300099b0 tx=0x7f1930009250 comp rx=0 tx=0).stop 2023-09-30T01:58:44.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.719+0000 7f194025c700 1 -- 172.21.15.7:0/2110336058 shutdown_connections 2023-09-30T01:58:44.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.719+0000 7f194025c700 1 --2- 172.21.15.7:0/2110336058 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f193813bb50 0x7f193813df30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:44.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.719+0000 7f194025c700 1 --2- 172.21.15.7:0/2110336058 >> v2:172.21.15.7:3300/0 conn(0x7f193806cde0 0x7f193806c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:44.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.719+0000 7f194025c700 1 -- 172.21.15.7:0/2110336058 >> 172.21.15.7:0/2110336058 conn(0x7f1938070cf0 msgr2=0x7f19380710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:44.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.719+0000 7f194025c700 1 -- 172.21.15.7:0/2110336058 shutdown_connections 2023-09-30T01:58:44.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.719+0000 7f194025c700 1 -- 172.21.15.7:0/2110336058 wait complete. 2023-09-30T01:58:44.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.719+0000 7f194025c700 10 monclient: build_initial_monmap 2023-09-30T01:58:44.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.719+0000 7f194025c700 10 monclient: monmap: 2023-09-30T01:58:44.731 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:44.732 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:58:44.732 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:58:44.732 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:58:44.732 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:44.732 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:44.732 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:58:44.732 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:58:44.732 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:58:44.732 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:44.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.719+0000 7f194025c700 1 Processor -- start 2023-09-30T01:58:44.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f194025c700 1 -- start start 2023-09-30T01:58:44.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f194025c700 10 monclient: init 2023-09-30T01:58:44.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f194025c700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:44.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f194025c700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:58:44.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f194025c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f193806cde0 0x7f19380a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:44.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f194025c700 10 monclient(hunting): picked mon.noname-c con 0x7f193806cde0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:44.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f194025c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19380a2170 0x7f193809c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:44.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f194025c700 10 monclient(hunting): picked mon.noname-b con 0x7f19380a2170 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:58:44.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f194025c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f193813bb50 0x7f193809cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:44.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f194025c700 10 monclient(hunting): picked mon.noname-a con 0x7f193813bb50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:58:44.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f194025c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:44.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f194025c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19380a12b0 con 0x7f193813bb50 2023-09-30T01:58:44.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f194025c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:44.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f194025c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1938137920 con 0x7f19380a2170 2023-09-30T01:58:44.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f194025c700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:44.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f194025c700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193809e840 con 0x7f193806cde0 2023-09-30T01:58:44.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f194025c700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:44.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f194025c700 10 monclient(hunting): authenticate will time out at 1413.785504s 2023-09-30T01:58:44.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193dff8700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f193806cde0 0x7f19380a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193e7f9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f193813bb50 0x7f193809cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193e7f9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f193813bb50 0x7f193809cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37650/0 (socket says 172.21.15.7:37650) 2023-09-30T01:58:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193dff8700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f193806cde0 0x7f19380a0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:50196/0 (socket says 172.21.15.7:50196) 2023-09-30T01:58:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193e7f9700 1 -- 172.21.15.7:0/3722078329 learned_addr learned my addr 172.21.15.7:0/3722078329 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193e7f9700 10 monclient(hunting): get_auth_request con 0x7f193813bb50 auth_method 0 2023-09-30T01:58:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193e7f9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193e7f9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193e7f9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193dff8700 10 monclient(hunting): get_auth_request con 0x7f193806cde0 auth_method 0 2023-09-30T01:58:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193dff8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193dff8700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:44.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193dff8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193e7f9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193e7f9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193e7f9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193dff8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193dff8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193dff8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193d7f7700 1 --2- 172.21.15.7:0/3722078329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19380a2170 0x7f193809c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193d7f7700 10 monclient(hunting): get_auth_request con 0x7f19380a2170 auth_method 0 2023-09-30T01:58:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193d7f7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193d7f7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:44.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193d7f7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:44.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193e7f9700 10 monclient(hunting): handle_auth_done global_id 4632 payload 824 2023-09-30T01:58:44.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193e7f9700 1 -- 172.21.15.7:0/3722078329 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f193806cde0 msgr2=0x7f19380a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:44.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193e7f9700 1 --2- 172.21.15.7:0/3722078329 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f193806cde0 0x7f19380a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:44.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193e7f9700 1 -- 172.21.15.7:0/3722078329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19380a2170 msgr2=0x7f193809c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:44.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193e7f9700 1 --2- 172.21.15.7:0/3722078329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19380a2170 0x7f193809c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:44.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193e7f9700 10 monclient: _finish_hunting 0 2023-09-30T01:58:44.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193e7f9700 1 monclient: found mon.noname-a 2023-09-30T01:58:44.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193e7f9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:44.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193e7f9700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:58:44.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.720+0000 7f193e7f9700 1 -- 172.21.15.7:0/3722078329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f192c009420 con 0x7f193813bb50 2023-09-30T01:58:44.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f193e7f9700 1 -- 172.21.15.7:0/3722078329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f193813bb50 msgr2=0x7f193809cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:44.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f193e7f9700 1 --2- 172.21.15.7:0/3722078329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f193813bb50 0x7f193809cc60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1930006fd0 tx=0x7f1930016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:44.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f1935ffb700 1 -- 172.21.15.7:0/3722078329 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1930006b80 con 0x7f193813bb50 2023-09-30T01:58:44.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f1935ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:44.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f1935ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:58:44.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f1935ffb700 10 monclient: dump: 2023-09-30T01:58:44.738 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:44.738 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:44.738 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:44.738 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:44.738 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:44.738 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:44.738 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:44.738 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:44.739 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:44.739 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:44.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f1935ffb700 10 monclient: _finish_auth 0 2023-09-30T01:58:44.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f1935ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:44.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f1935ffb700 1 -- 172.21.15.7:0/3722078329 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193001c460 con 0x7f193813bb50 2023-09-30T01:58:44.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f194025c700 5 monclient: authenticate success, global_id 4632 2023-09-30T01:58:44.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f194025c700 10 monclient: _renew_subs 2023-09-30T01:58:44.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f194025c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:44.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f194025c700 1 -- 172.21.15.7:0/3722078329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19300094f0 con 0x7f193813bb50 2023-09-30T01:58:44.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f1935ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:44.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f1935ffb700 1 -- 172.21.15.7:0/3722078329 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193001c820 con 0x7f193813bb50 2023-09-30T01:58:44.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f1935ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:44.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f1935ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:58:44.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f1935ffb700 10 monclient: dump: 2023-09-30T01:58:44.740 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:44.740 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:44.740 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:44.740 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:44.740 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:44.740 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:44.740 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:44.740 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:44.740 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:44.740 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:44.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f194025c700 10 monclient: _renew_subs 2023-09-30T01:58:44.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f1937fff700 4 set_mon_vals no callback set 2023-09-30T01:58:44.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f194025c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:44.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.723+0000 7f194025c700 1 -- 172.21.15.7:0/3722078329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193809fdf0 con 0x7f193813bb50 2023-09-30T01:58:44.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.724+0000 7f194025c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:44.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.724+0000 7f194025c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:44.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.724+0000 7f194025c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:44.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.724+0000 7f1935ffb700 1 -- 172.21.15.7:0/3722078329 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f193001cc10 con 0x7f193813bb50 2023-09-30T01:58:44.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.724+0000 7f194025c700 1 -- 172.21.15.7:0/3722078329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f193809efc0 con 0x7f193813bb50 2023-09-30T01:58:44.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.724+0000 7f1935ffb700 1 --2- 172.21.15.7:0/3722078329 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f19240357e0 0x7f1924037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:44.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.724+0000 7f1935ffb700 1 -- 172.21.15.7:0/3722078329 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f193004a1a0 con 0x7f193813bb50 2023-09-30T01:58:44.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.725+0000 7f193dff8700 1 --2- 172.21.15.7:0/3722078329 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f19240357e0 0x7f1924037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:44.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.725+0000 7f193dff8700 10 monclient: get_auth_request con 0x7f19240357e0 auth_method 0 2023-09-30T01:58:44.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.727+0000 7f193dff8700 1 --2- 172.21.15.7:0/3722078329 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f19240357e0 0x7f1924037c90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1928007fd0 tx=0x7f19280026a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:44.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.729+0000 7f1935ffb700 1 -- 172.21.15.7:0/3722078329 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f193809efc0 con 0x7f193813bb50 2023-09-30T01:58:44.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.729+0000 7f1935ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:44.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.729+0000 7f1935ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:58:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.967+0000 7f194025c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:58:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.967+0000 7f194025c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.967+0000 7f194025c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.967+0000 7f194025c700 1 -- 172.21.15.7:0/3722078329 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f193809efc0 con 0x7f193813bb50 2023-09-30T01:58:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.967+0000 7f1935ffb700 1 -- 172.21.15.7:0/3722078329 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f193809efc0 con 0x7f193813bb50 2023-09-30T01:58:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.967+0000 7f1935ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:44.967 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.967+0000 7f1935ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:58:44.968 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:58:44.968 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:44.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.969+0000 7f194025c700 1 -- 172.21.15.7:0/3722078329 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f19240357e0 msgr2=0x7f1924037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:44.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.969+0000 7f194025c700 1 --2- 172.21.15.7:0/3722078329 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f19240357e0 0x7f1924037c90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1928007fd0 tx=0x7f19280026a0 comp rx=0 tx=0).stop 2023-09-30T01:58:44.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.969+0000 7f194025c700 10 monclient: shutdown 2023-09-30T01:58:44.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.969+0000 7f194025c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:44.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.969+0000 7f194025c700 1 -- 172.21.15.7:0/3722078329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f193813bb50 msgr2=0x7f193809cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:44.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.969+0000 7f194025c700 1 --2- 172.21.15.7:0/3722078329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f193813bb50 0x7f193809cc60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1930006fd0 tx=0x7f1930016000 comp rx=0 tx=0).stop 2023-09-30T01:58:44.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.969+0000 7f194025c700 1 -- 172.21.15.7:0/3722078329 shutdown_connections 2023-09-30T01:58:44.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.969+0000 7f194025c700 1 --2- 172.21.15.7:0/3722078329 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f19240357e0 0x7f1924037c90 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:44.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.969+0000 7f194025c700 1 --2- 172.21.15.7:0/3722078329 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f193813bb50 0x7f193809cc60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:44.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.969+0000 7f194025c700 1 --2- 172.21.15.7:0/3722078329 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f193806cde0 0x7f19380a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:44.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.969+0000 7f194025c700 1 --2- 172.21.15.7:0/3722078329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19380a2170 0x7f193809c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:44.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.969+0000 7f194025c700 1 -- 172.21.15.7:0/3722078329 >> 172.21.15.7:0/3722078329 conn(0x7f1938070cf0 msgr2=0x7f19380a3670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:44.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.969+0000 7f194025c700 1 -- 172.21.15.7:0/3722078329 shutdown_connections 2023-09-30T01:58:44.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:44.969+0000 7f194025c700 1 -- 172.21.15.7:0/3722078329 wait complete. 2023-09-30T01:58:47.981 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:48.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.112+0000 7f8d64c43700 10 monclient: get_monmap_and_config 2023-09-30T01:58:48.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.112+0000 7f8d64c43700 10 monclient: build_initial_monmap 2023-09-30T01:58:48.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.112+0000 7f8d64c43700 10 monclient: monmap: 2023-09-30T01:58:48.113 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:48.113 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:48.113 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:58:48.113748+0000 2023-09-30T01:58:48.113 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:58:48.113748+0000 2023-09-30T01:58:48.113 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:48.114 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:48.114 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:58:48.114 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:58:48.114 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:58:48.114 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:58:48.114 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:58:48.114 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:48.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.113+0000 7f8d64c43700 1 Processor -- start 2023-09-30T01:58:48.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.113+0000 7f8d64c43700 1 -- start start 2023-09-30T01:58:48.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.113+0000 7f8d64c43700 10 monclient: init 2023-09-30T01:58:48.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d64c43700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:48.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d64c43700 10 monclient: _add_conns ranks=[3,4,0,2,1] 2023-09-30T01:58:48.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d64c43700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8d60098540 0x7f8d601329e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:48.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d64c43700 10 monclient(hunting): picked mon.noname-a con 0x7f8d60098540 addr v2:172.21.15.7:3300/0 2023-09-30T01:58:48.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d64c43700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8d60098910 0x7f8d60098d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:48.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d64c43700 10 monclient(hunting): picked mon.noname-c con 0x7f8d60098910 addr v2:172.21.15.50:3300/0 2023-09-30T01:58:48.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d64c43700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8d6012ebc0 addr v1:172.21.15.7:6789/0 2023-09-30T01:58:48.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d64c43700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d60069f10 con 0x7f8d6012ebc0 2023-09-30T01:58:48.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d64c43700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:48.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d64c43700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d600a0f40 con 0x7f8d60098540 2023-09-30T01:58:48.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d64c43700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:48.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d64c43700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d60132f20 con 0x7f8d60098910 2023-09-30T01:58:48.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d64c43700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:48.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d64c43700 10 monclient(hunting): authenticate will time out at 1417.179177s 2023-09-30T01:58:48.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8d60098540 0x7f8d601329e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:48.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5ffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8d6012ebc0 0x7f8d6012efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46492/0 (socket says 172.21.15.7:46492) 2023-09-30T01:58:48.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5ffff700 1 -- 172.21.15.7:0/4204464819 learned_addr learned my addr 172.21.15.7:0/4204464819 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:58:48.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5f7fe700 10 monclient(hunting): get_auth_request con 0x7f8d60098540 auth_method 0 2023-09-30T01:58:48.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:48.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5effd700 1 --2- 172.21.15.7:0/4204464819 >> v2:172.21.15.50:3300/0 conn(0x7f8d60098910 0x7f8d60098d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5effd700 10 monclient(hunting): get_auth_request con 0x7f8d60098910 auth_method 0 2023-09-30T01:58:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5f7fe700 10 monclient(hunting): handle_auth_done global_id 4635 payload 274 2023-09-30T01:58:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5f7fe700 1 -- 172.21.15.7:0/4204464819 >> v2:172.21.15.50:3300/0 conn(0x7f8d60098910 msgr2=0x7f8d60098d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:48.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5f7fe700 1 --2- 172.21.15.7:0/4204464819 >> v2:172.21.15.50:3300/0 conn(0x7f8d60098910 0x7f8d60098d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:48.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5f7fe700 1 -- 172.21.15.7:0/4204464819 >> v1:172.21.15.7:6789/0 conn(0x7f8d6012ebc0 legacy=0x7f8d6012efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:48.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5f7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:58:48.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5f7fe700 1 monclient: found mon.noname-a 2023-09-30T01:58:48.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:48.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:58:48.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5f7fe700 1 -- 172.21.15.7:0/4204464819 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d60096a50 con 0x7f8d60098540 2023-09-30T01:58:48.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.114+0000 7f8d5effd700 1 --2- 172.21.15.7:0/4204464819 >> v2:172.21.15.50:3300/0 conn(0x7f8d60098910 0x7f8d60098d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:58:48.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.116+0000 7f8d5f7fe700 1 -- 172.21.15.7:0/4204464819 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d60098540 msgr2=0x7f8d601329e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:48.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.116+0000 7f8d5f7fe700 1 --2- 172.21.15.7:0/4204464819 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d60098540 0x7f8d601329e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8d6006b680 tx=0x7f8d500091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39e13bb3fca91cb3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:48.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d5e7fc700 1 -- 172.21.15.7:0/4204464819 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5000b880 con 0x7f8d60098540 2023-09-30T01:58:48.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d5e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:48.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d5e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:58:48.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d5e7fc700 10 monclient: dump: 2023-09-30T01:58:48.123 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:48.123 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:48.123 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:48.123 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:48.123 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:48.123 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:48.123 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:48.123 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:48.123 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:48.124 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d5e7fc700 10 monclient: _finish_auth 0 2023-09-30T01:58:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d5e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d5e7fc700 1 -- 172.21.15.7:0/4204464819 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5000b9e0 con 0x7f8d60098540 2023-09-30T01:58:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d64c43700 5 monclient: authenticate success, global_id 4635 2023-09-30T01:58:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d64c43700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:58:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d5e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d5e7fc700 1 -- 172.21.15.7:0/4204464819 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5000bd80 con 0x7f8d60098540 2023-09-30T01:58:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d64c43700 10 monclient: get_monmap_and_config success 2023-09-30T01:58:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d64c43700 4 set_mon_vals no callback set 2023-09-30T01:58:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d5e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:48.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d5e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:58:48.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d5e7fc700 10 monclient: dump: 2023-09-30T01:58:48.125 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:48.125 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:48.125 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:48.125 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:48.125 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:48.125 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:48.125 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:48.125 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:48.125 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:48.126 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:48.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d64c43700 10 monclient: shutdown 2023-09-30T01:58:48.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d64c43700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:48.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d64c43700 1 -- 172.21.15.7:0/4204464819 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d60098540 msgr2=0x7f8d601329e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:48.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d64c43700 1 --2- 172.21.15.7:0/4204464819 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d60098540 0x7f8d601329e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8d6006b680 tx=0x7f8d500091a0 comp rx=0 tx=0).stop 2023-09-30T01:58:48.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d64c43700 1 -- 172.21.15.7:0/4204464819 shutdown_connections 2023-09-30T01:58:48.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d64c43700 1 --2- 172.21.15.7:0/4204464819 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d60098540 0x7f8d601329e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:48.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d64c43700 1 --2- 172.21.15.7:0/4204464819 >> v2:172.21.15.50:3300/0 conn(0x7f8d60098910 0x7f8d60098d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:48.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d64c43700 1 -- 172.21.15.7:0/4204464819 >> 172.21.15.7:0/4204464819 conn(0x7f8d60071830 msgr2=0x7f8d600a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:48.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d64c43700 1 -- 172.21.15.7:0/4204464819 shutdown_connections 2023-09-30T01:58:48.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.117+0000 7f8d64c43700 1 -- 172.21.15.7:0/4204464819 wait complete. 2023-09-30T01:58:48.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.118+0000 7f8d64c43700 10 monclient: build_initial_monmap 2023-09-30T01:58:48.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.118+0000 7f8d64c43700 10 monclient: monmap: 2023-09-30T01:58:48.127 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:48.127 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:58:48.127 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:58:48.127 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:58:48.127 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:48.127 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:48.127 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:58:48.127 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:58:48.127 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:58:48.128 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:48.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.118+0000 7f8d64c43700 1 Processor -- start 2023-09-30T01:58:48.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.118+0000 7f8d64c43700 1 -- start start 2023-09-30T01:58:48.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.118+0000 7f8d64c43700 10 monclient: init 2023-09-30T01:58:48.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d64c43700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:48.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d64c43700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:58:48.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d64c43700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d60098540 0x7f8d6011ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:48.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d64c43700 10 monclient(hunting): picked mon.noname-a con 0x7f8d60098540 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:58:48.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d64c43700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8d60098910 0x7f8d6011bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:48.133 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d64c43700 10 monclient(hunting): picked mon.noname-c con 0x7f8d60098910 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:48.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d64c43700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d6012ebc0 0x7f8d6011c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:48.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d64c43700 10 monclient(hunting): picked mon.noname-b con 0x7f8d6012ebc0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:58:48.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d64c43700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:48.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d64c43700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d601327b0 con 0x7f8d60098540 2023-09-30T01:58:48.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d64c43700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:48.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d64c43700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d600a4bc0 con 0x7f8d6012ebc0 2023-09-30T01:58:48.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d64c43700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:48.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d64c43700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6011ea40 con 0x7f8d60098910 2023-09-30T01:58:48.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d64c43700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:48.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d64c43700 10 monclient(hunting): authenticate will time out at 1417.184356s 2023-09-30T01:58:48.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d60098540 0x7f8d6011ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:48.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d60098540 0x7f8d6011ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37670/0 (socket says 172.21.15.7:37670) 2023-09-30T01:58:48.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5f7fe700 1 -- 172.21.15.7:0/324083625 learned_addr learned my addr 172.21.15.7:0/324083625 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:48.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5f7fe700 10 monclient(hunting): get_auth_request con 0x7f8d60098540 auth_method 0 2023-09-30T01:58:48.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:48.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:48.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:48.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5effd700 1 --2- 172.21.15.7:0/324083625 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8d60098910 0x7f8d6011bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:48.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5effd700 10 monclient(hunting): get_auth_request con 0x7f8d60098910 auth_method 0 2023-09-30T01:58:48.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:48.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:48.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:48.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:48.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:48.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:48.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:48.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:48.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:48.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5ffff700 1 --2- 172.21.15.7:0/324083625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d6012ebc0 0x7f8d6011c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:48.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5ffff700 10 monclient(hunting): get_auth_request con 0x7f8d6012ebc0 auth_method 0 2023-09-30T01:58:48.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:48.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:48.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:48.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5f7fe700 10 monclient(hunting): handle_auth_done global_id 4638 payload 824 2023-09-30T01:58:48.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5f7fe700 1 -- 172.21.15.7:0/324083625 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8d60098910 msgr2=0x7f8d6011bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:48.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5f7fe700 1 --2- 172.21.15.7:0/324083625 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8d60098910 0x7f8d6011bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:48.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5f7fe700 1 -- 172.21.15.7:0/324083625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d6012ebc0 msgr2=0x7f8d6011c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:48.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5f7fe700 1 --2- 172.21.15.7:0/324083625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d6012ebc0 0x7f8d6011c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:48.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5f7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:58:48.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5f7fe700 1 monclient: found mon.noname-a 2023-09-30T01:58:48.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:48.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:58:48.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.119+0000 7f8d5f7fe700 1 -- 172.21.15.7:0/324083625 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d50009420 con 0x7f8d60098540 2023-09-30T01:58:48.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d5f7fe700 1 -- 172.21.15.7:0/324083625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d60098540 msgr2=0x7f8d6011ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:48.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d5f7fe700 1 --2- 172.21.15.7:0/324083625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d60098540 0x7f8d6011ba20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000c010 tx=0x7f8d50016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:48.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d477fe700 1 -- 172.21.15.7:0/324083625 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d50006af0 con 0x7f8d60098540 2023-09-30T01:58:48.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:48.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d477fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:58:48.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d477fe700 10 monclient: dump: 2023-09-30T01:58:48.138 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:48.138 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:48.138 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:48.138 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:48.138 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:48.138 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:48.139 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:48.139 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:48.139 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:48.139 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:48.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d477fe700 10 monclient: _finish_auth 0 2023-09-30T01:58:48.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d477fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:48.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d64c43700 5 monclient: authenticate success, global_id 4638 2023-09-30T01:58:48.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d477fe700 1 -- 172.21.15.7:0/324083625 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d50020070 con 0x7f8d60098540 2023-09-30T01:58:48.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d64c43700 10 monclient: _renew_subs 2023-09-30T01:58:48.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d64c43700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:48.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d64c43700 1 -- 172.21.15.7:0/324083625 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d600a05c0 con 0x7f8d60098540 2023-09-30T01:58:48.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d477fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:48.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d477fe700 1 -- 172.21.15.7:0/324083625 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d50000c80 con 0x7f8d60098540 2023-09-30T01:58:48.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:48.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d477fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:58:48.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d477fe700 10 monclient: dump: 2023-09-30T01:58:48.140 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:48.140 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:48.140 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:48.140 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:48.140 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:48.140 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:48.141 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:48.141 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:48.141 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:48.141 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:48.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d5dffb700 4 set_mon_vals no callback set 2023-09-30T01:58:48.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d64c43700 10 monclient: _renew_subs 2023-09-30T01:58:48.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d64c43700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:48.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.122+0000 7f8d64c43700 1 -- 172.21.15.7:0/324083625 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d6011cc30 con 0x7f8d60098540 2023-09-30T01:58:48.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.123+0000 7f8d477fe700 1 -- 172.21.15.7:0/324083625 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f8d50022030 con 0x7f8d60098540 2023-09-30T01:58:48.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.123+0000 7f8d64c43700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:48.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.123+0000 7f8d64c43700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:48.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.123+0000 7f8d64c43700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:48.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.123+0000 7f8d477fe700 1 --2- 172.21.15.7:0/324083625 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f8d4c0357e0 0x7f8d4c037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:48.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.123+0000 7f8d64c43700 1 -- 172.21.15.7:0/324083625 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d6011ec00 con 0x7f8d60098540 2023-09-30T01:58:48.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.123+0000 7f8d477fe700 1 -- 172.21.15.7:0/324083625 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f8d50049940 con 0x7f8d60098540 2023-09-30T01:58:48.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.123+0000 7f8d5effd700 1 --2- 172.21.15.7:0/324083625 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f8d4c0357e0 0x7f8d4c037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:48.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.123+0000 7f8d5effd700 10 monclient: get_auth_request con 0x7f8d4c0357e0 auth_method 0 2023-09-30T01:58:48.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.126+0000 7f8d5effd700 1 --2- 172.21.15.7:0/324083625 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f8d4c0357e0 0x7f8d4c037c90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8d54006f00 tx=0x7f8d54008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:48.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.127+0000 7f8d477fe700 1 -- 172.21.15.7:0/324083625 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8d5001a9c0 con 0x7f8d60098540 2023-09-30T01:58:48.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.127+0000 7f8d477fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:48.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.127+0000 7f8d477fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:58:48.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.367+0000 7f8d64c43700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:58:48.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.367+0000 7f8d64c43700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:48.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.367+0000 7f8d64c43700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:48.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.367+0000 7f8d64c43700 1 -- 172.21.15.7:0/324083625 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8d6011ec00 con 0x7f8d60098540 2023-09-30T01:58:48.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.369+0000 7f8d477fe700 1 -- 172.21.15.7:0/324083625 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8d5003c980 con 0x7f8d60098540 2023-09-30T01:58:48.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.369+0000 7f8d477fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:48.369 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.369+0000 7f8d477fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:58:48.369 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:58:48.370 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:48.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.371+0000 7f8d64c43700 1 -- 172.21.15.7:0/324083625 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f8d4c0357e0 msgr2=0x7f8d4c037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:48.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.371+0000 7f8d64c43700 1 --2- 172.21.15.7:0/324083625 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f8d4c0357e0 0x7f8d4c037c90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8d54006f00 tx=0x7f8d54008040 comp rx=0 tx=0).stop 2023-09-30T01:58:48.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.371+0000 7f8d64c43700 10 monclient: shutdown 2023-09-30T01:58:48.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.371+0000 7f8d64c43700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:48.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.371+0000 7f8d64c43700 1 -- 172.21.15.7:0/324083625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d60098540 msgr2=0x7f8d6011ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:48.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.371+0000 7f8d64c43700 1 --2- 172.21.15.7:0/324083625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d60098540 0x7f8d6011ba20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000c010 tx=0x7f8d50016000 comp rx=0 tx=0).stop 2023-09-30T01:58:48.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.371+0000 7f8d64c43700 1 -- 172.21.15.7:0/324083625 shutdown_connections 2023-09-30T01:58:48.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.371+0000 7f8d64c43700 1 --2- 172.21.15.7:0/324083625 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f8d4c0357e0 0x7f8d4c037c90 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:48.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.371+0000 7f8d64c43700 1 --2- 172.21.15.7:0/324083625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d60098540 0x7f8d6011ba20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:48.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.371+0000 7f8d64c43700 1 --2- 172.21.15.7:0/324083625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8d6012ebc0 0x7f8d6011c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:48.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.371+0000 7f8d64c43700 1 --2- 172.21.15.7:0/324083625 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8d60098910 0x7f8d6011bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:48.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.371+0000 7f8d64c43700 1 -- 172.21.15.7:0/324083625 >> 172.21.15.7:0/324083625 conn(0x7f8d60071830 msgr2=0x7f8d60131940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:48.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.371+0000 7f8d64c43700 1 -- 172.21.15.7:0/324083625 shutdown_connections 2023-09-30T01:58:48.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:48.371+0000 7f8d64c43700 1 -- 172.21.15.7:0/324083625 wait complete. 2023-09-30T01:58:51.384 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:51.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.516+0000 7f319b21a700 10 monclient: get_monmap_and_config 2023-09-30T01:58:51.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.516+0000 7f319b21a700 10 monclient: build_initial_monmap 2023-09-30T01:58:51.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.516+0000 7f319b21a700 10 monclient: monmap: 2023-09-30T01:58:51.517 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:51.517 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:51.517 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:58:51.517658+0000 2023-09-30T01:58:51.517 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:58:51.517658+0000 2023-09-30T01:58:51.517 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:51.517 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:51.517 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:58:51.517 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:58:51.517 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:58:51.517 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:58:51.517 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:58:51.518 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:51.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.517+0000 7f319b21a700 1 Processor -- start 2023-09-30T01:58:51.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.517+0000 7f319b21a700 1 -- start start 2023-09-30T01:58:51.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.517+0000 7f319b21a700 10 monclient: init 2023-09-30T01:58:51.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.517+0000 7f319b21a700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:51.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.517+0000 7f319b21a700 10 monclient: _add_conns ranks=[2,3,4,0,1] 2023-09-30T01:58:51.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.517+0000 7f319b21a700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3194096a50 0x7f31941327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:51.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.517+0000 7f319b21a700 10 monclient(hunting): picked mon.noname-b con 0x7f3194096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:51.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.517+0000 7f319b21a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f31940a2390 0x7f3194096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:51.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.517+0000 7f319b21a700 10 monclient(hunting): picked mon.noname-a con 0x7f31940a2390 addr v2:172.21.15.7:3300/0 2023-09-30T01:58:51.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.517+0000 7f319b21a700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3194097360 0x7f319412ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:51.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.517+0000 7f319b21a700 10 monclient(hunting): picked mon.noname-c con 0x7f3194097360 addr v2:172.21.15.50:3300/0 2023-09-30T01:58:51.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.517+0000 7f319b21a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:51.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.517+0000 7f319b21a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31940980e0 con 0x7f31940a2390 2023-09-30T01:58:51.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.517+0000 7f319b21a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:51.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.517+0000 7f319b21a700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31940991a0 con 0x7f3194097360 2023-09-30T01:58:51.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.517+0000 7f319b21a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:51.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.517+0000 7f319b21a700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3194099480 con 0x7f3194096a50 2023-09-30T01:58:51.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.517+0000 7f319b21a700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:51.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.517+0000 7f319b21a700 10 monclient(hunting): authenticate will time out at 1420.582942s 2023-09-30T01:58:51.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3198fb6700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3194096a50 0x7f31941327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:51.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3193fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f31940a2390 0x7f3194096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:51.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3193fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f31940a2390 0x7f3194096e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37682/0 (socket says 172.21.15.7:37682) 2023-09-30T01:58:51.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3198fb6700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3194096a50 0x7f31941327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:50224/0 (socket says 172.21.15.7:50224) 2023-09-30T01:58:51.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3193fff700 1 -- 172.21.15.7:0/1388874736 learned_addr learned my addr 172.21.15.7:0/1388874736 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:51.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3193fff700 10 monclient(hunting): get_auth_request con 0x7f31940a2390 auth_method 0 2023-09-30T01:58:51.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3193fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:51.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3193fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:51.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3193fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:51.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3198fb6700 10 monclient(hunting): get_auth_request con 0x7f3194096a50 auth_method 0 2023-09-30T01:58:51.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3198fb6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:51.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3198fb6700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:51.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3198fb6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:51.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3193fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:51.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3193fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:51.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3193fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:51.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3198fb6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:51.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3198fb6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:51.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3198fb6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:51.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3193fff700 10 monclient(hunting): handle_auth_done global_id 4641 payload 274 2023-09-30T01:58:51.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f31997b7700 1 --2- 172.21.15.7:0/1388874736 >> v2:172.21.15.50:3300/0 conn(0x7f3194097360 0x7f319412ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:51.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3193fff700 1 -- 172.21.15.7:0/1388874736 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3194096a50 msgr2=0x7f31941327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:51.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3193fff700 1 --2- 172.21.15.7:0/1388874736 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3194096a50 0x7f31941327a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3193fff700 1 -- 172.21.15.7:0/1388874736 >> v2:172.21.15.50:3300/0 conn(0x7f3194097360 msgr2=0x7f319412ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3193fff700 1 --2- 172.21.15.7:0/1388874736 >> v2:172.21.15.50:3300/0 conn(0x7f3194097360 0x7f319412ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3193fff700 10 monclient: _finish_hunting 0 2023-09-30T01:58:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3193fff700 1 monclient: found mon.noname-a 2023-09-30T01:58:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3193fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3193fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:58:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3193fff700 1 -- 172.21.15.7:0/1388874736 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3194097bf0 con 0x7f31940a2390 2023-09-30T01:58:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f31997b7700 10 monclient: get_auth_request con 0x7f3194097360 auth_method 0 2023-09-30T01:58:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f3198fb6700 1 --2- 172.21.15.7:0/1388874736 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3194096a50 0x7f31941327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:58:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.518+0000 7f31997b7700 1 --2- 172.21.15.7:0/1388874736 >> v2:172.21.15.50:3300/0 conn(0x7f3194097360 0x7f319412ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:58:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.520+0000 7f3193fff700 1 -- 172.21.15.7:0/1388874736 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31940a2390 msgr2=0x7f3194096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.520+0000 7f3193fff700 1 --2- 172.21.15.7:0/1388874736 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31940a2390 0x7f3194096e20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f31940a2010 tx=0x7f3180009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94bd47d15a795b2c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.520+0000 7f31937fe700 1 -- 172.21.15.7:0/1388874736 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3180009e70 con 0x7f31940a2390 2023-09-30T01:58:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.520+0000 7f31937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.520+0000 7f31937fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:58:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.520+0000 7f31937fe700 10 monclient: dump: 2023-09-30T01:58:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:51.528 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.520+0000 7f31937fe700 10 monclient: _finish_auth 0 2023-09-30T01:58:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.520+0000 7f31937fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.520+0000 7f31937fe700 1 -- 172.21.15.7:0/1388874736 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f318000ba70 con 0x7f31940a2390 2023-09-30T01:58:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.520+0000 7f319b21a700 5 monclient: authenticate success, global_id 4641 2023-09-30T01:58:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.520+0000 7f319b21a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:58:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.520+0000 7f31937fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.520+0000 7f319b21a700 10 monclient: get_monmap_and_config success 2023-09-30T01:58:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.520+0000 7f31937fe700 1 -- 172.21.15.7:0/1388874736 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318000be30 con 0x7f31940a2390 2023-09-30T01:58:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.520+0000 7f31937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.520+0000 7f319b21a700 4 set_mon_vals no callback set 2023-09-30T01:58:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.520+0000 7f31937fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:58:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.520+0000 7f31937fe700 10 monclient: dump: 2023-09-30T01:58:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:51.530 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.521+0000 7f319b21a700 10 monclient: shutdown 2023-09-30T01:58:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.521+0000 7f319b21a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.521+0000 7f319b21a700 1 -- 172.21.15.7:0/1388874736 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31940a2390 msgr2=0x7f3194096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.521+0000 7f319b21a700 1 --2- 172.21.15.7:0/1388874736 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31940a2390 0x7f3194096e20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f31940a2010 tx=0x7f3180009250 comp rx=0 tx=0).stop 2023-09-30T01:58:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.521+0000 7f319b21a700 1 -- 172.21.15.7:0/1388874736 shutdown_connections 2023-09-30T01:58:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.521+0000 7f319b21a700 1 --2- 172.21.15.7:0/1388874736 >> v2:172.21.15.50:3300/0 conn(0x7f3194097360 0x7f319412ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.521+0000 7f319b21a700 1 --2- 172.21.15.7:0/1388874736 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3194096a50 0x7f31941327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.521+0000 7f319b21a700 1 --2- 172.21.15.7:0/1388874736 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31940a2390 0x7f3194096e20 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.521+0000 7f319b21a700 1 -- 172.21.15.7:0/1388874736 >> 172.21.15.7:0/1388874736 conn(0x7f3194070cf0 msgr2=0x7f31940710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.521+0000 7f319b21a700 1 -- 172.21.15.7:0/1388874736 shutdown_connections 2023-09-30T01:58:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.521+0000 7f319b21a700 1 -- 172.21.15.7:0/1388874736 wait complete. 2023-09-30T01:58:51.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.521+0000 7f319b21a700 10 monclient: build_initial_monmap 2023-09-30T01:58:51.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.521+0000 7f319b21a700 10 monclient: monmap: 2023-09-30T01:58:51.531 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:51.531 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:58:51.531 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:58:51.531 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:58:51.531 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:51.531 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:51.531 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:58:51.531 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:58:51.531 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:58:51.531 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:51.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.522+0000 7f319b21a700 1 Processor -- start 2023-09-30T01:58:51.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.522+0000 7f319b21a700 1 -- start start 2023-09-30T01:58:51.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.522+0000 7f319b21a700 10 monclient: init 2023-09-30T01:58:51.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f319b21a700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:51.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f319b21a700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:58:51.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f319b21a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3194096a50 0x7f319412df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:51.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f319b21a700 10 monclient(hunting): picked mon.noname-b con 0x7f3194096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:58:51.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f319b21a700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3194097360 0x7f319409db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:51.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f319b21a700 10 monclient(hunting): picked mon.noname-c con 0x7f3194097360 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:51.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f319b21a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31940a2390 0x7f319409e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:51.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f319b21a700 10 monclient(hunting): picked mon.noname-a con 0x7f31940a2390 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:58:51.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f319b21a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:51.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f319b21a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31940a1960 con 0x7f31940a2390 2023-09-30T01:58:51.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f319b21a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:51.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f319b21a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3194133580 con 0x7f3194096a50 2023-09-30T01:58:51.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f319b21a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:51.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f319b21a700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f319412ed30 con 0x7f3194097360 2023-09-30T01:58:51.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f319b21a700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:51.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f319b21a700 10 monclient(hunting): authenticate will time out at 1420.588146s 2023-09-30T01:58:51.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3193fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3194097360 0x7f319409db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:51.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3193fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3194097360 0x7f319409db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:50240/0 (socket says 172.21.15.7:50240) 2023-09-30T01:58:51.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3193fff700 1 -- 172.21.15.7:0/3973252313 learned_addr learned my addr 172.21.15.7:0/3973252313 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:51.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3193fff700 10 monclient(hunting): get_auth_request con 0x7f3194097360 auth_method 0 2023-09-30T01:58:51.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3193fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:51.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3193fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:51.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3193fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:51.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3198fb6700 1 --2- 172.21.15.7:0/3973252313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3194096a50 0x7f319412df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:51.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3198fb6700 10 monclient(hunting): get_auth_request con 0x7f3194096a50 auth_method 0 2023-09-30T01:58:51.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f31997b7700 1 --2- 172.21.15.7:0/3973252313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31940a2390 0x7f319409e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:51.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3198fb6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:51.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3198fb6700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:51.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3198fb6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:51.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f31997b7700 10 monclient(hunting): get_auth_request con 0x7f31940a2390 auth_method 0 2023-09-30T01:58:51.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f31997b7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:51.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f31997b7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:51.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f31997b7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:51.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3193fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:51.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3193fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:51.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3193fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:51.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f31997b7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:51.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f31997b7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:51.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f31997b7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:51.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3198fb6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:51.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3198fb6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:51.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3198fb6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:51.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3193fff700 10 monclient(hunting): handle_auth_done global_id 4661 payload 824 2023-09-30T01:58:51.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3193fff700 1 -- 172.21.15.7:0/3973252313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3194096a50 msgr2=0x7f319412df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:51.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3193fff700 1 --2- 172.21.15.7:0/3973252313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3194096a50 0x7f319412df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:51.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3193fff700 1 -- 172.21.15.7:0/3973252313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31940a2390 msgr2=0x7f319409e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:51.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3193fff700 1 --2- 172.21.15.7:0/3973252313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31940a2390 0x7f319409e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:51.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3193fff700 10 monclient: _finish_hunting 0 2023-09-30T01:58:51.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3193fff700 1 monclient: found mon.noname-c 2023-09-30T01:58:51.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3193fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:51.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3193fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:58:51.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.523+0000 7f3193fff700 1 -- 172.21.15.7:0/3973252313 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3188009420 con 0x7f3194097360 2023-09-30T01:58:51.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.525+0000 7f3193fff700 1 -- 172.21.15.7:0/3973252313 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3194097360 msgr2=0x7f319409db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:51.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.525+0000 7f3193fff700 1 --2- 172.21.15.7:0/3973252313 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3194097360 0x7f319409db90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3180006fd0 tx=0x7f3180017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:51.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.525+0000 7f3190ff9700 1 -- 172.21.15.7:0/3973252313 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318001f070 con 0x7f3194097360 2023-09-30T01:58:51.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f3190ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:51.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f3190ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:58:51.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f3190ff9700 10 monclient: dump: 2023-09-30T01:58:51.541 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:51.541 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:51.541 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:51.541 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:51.541 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:51.541 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:51.542 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:51.542 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:51.542 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:51.542 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:51.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f3190ff9700 10 monclient: _finish_auth 0 2023-09-30T01:58:51.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f3190ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:51.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f319b21a700 5 monclient: authenticate success, global_id 4661 2023-09-30T01:58:51.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f3190ff9700 1 -- 172.21.15.7:0/3973252313 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3180006be0 con 0x7f3194097360 2023-09-30T01:58:51.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f319b21a700 10 monclient: _renew_subs 2023-09-30T01:58:51.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f319b21a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:51.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f319b21a700 1 -- 172.21.15.7:0/3973252313 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31800094f0 con 0x7f3194097360 2023-09-30T01:58:51.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f3190ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:51.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f3190ff9700 1 -- 172.21.15.7:0/3973252313 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3180011a50 con 0x7f3194097360 2023-09-30T01:58:51.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f3190ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:51.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f3190ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:58:51.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f3190ff9700 10 monclient: dump: 2023-09-30T01:58:51.543 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:51.543 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:51.543 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:51.543 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:51.543 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:51.543 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:51.543 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:51.543 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:51.544 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:51.544 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:51.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f3192ffd700 4 set_mon_vals no callback set 2023-09-30T01:58:51.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f319b21a700 10 monclient: _renew_subs 2023-09-30T01:58:51.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f319b21a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:51.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f319b21a700 1 -- 172.21.15.7:0/3973252313 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31940a1320 con 0x7f3194097360 2023-09-30T01:58:51.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f3190ff9700 1 -- 172.21.15.7:0/3973252313 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f3180023070 con 0x7f3194097360 2023-09-30T01:58:51.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f319b21a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:51.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f319b21a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:51.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f319b21a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:51.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.526+0000 7f319b21a700 1 -- 172.21.15.7:0/3973252313 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3194097160 con 0x7f3194097360 2023-09-30T01:58:51.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.527+0000 7f3190ff9700 1 --2- 172.21.15.7:0/3973252313 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3178035810 0x7f3178037cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:51.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.527+0000 7f3190ff9700 1 -- 172.21.15.7:0/3973252313 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f3180048e00 con 0x7f3194097360 2023-09-30T01:58:51.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.527+0000 7f3198fb6700 1 --2- 172.21.15.7:0/3973252313 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3178035810 0x7f3178037cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:51.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.527+0000 7f3198fb6700 10 monclient: get_auth_request con 0x7f3178035810 auth_method 0 2023-09-30T01:58:51.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.529+0000 7f3198fb6700 1 --2- 172.21.15.7:0/3973252313 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3178035810 0x7f3178037cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3188009eb0 tx=0x7f31880091a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:51.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.531+0000 7f3190ff9700 1 -- 172.21.15.7:0/3973252313 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3180011e40 con 0x7f3194097360 2023-09-30T01:58:51.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.531+0000 7f3190ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:51.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.531+0000 7f3190ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:58:51.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.770+0000 7f319b21a700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:58:51.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.770+0000 7f319b21a700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:51.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.770+0000 7f319b21a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:51.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.770+0000 7f319b21a700 1 -- 172.21.15.7:0/3973252313 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f31940710c0 con 0x7f3194097360 2023-09-30T01:58:51.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.771+0000 7f3190ff9700 1 -- 172.21.15.7:0/3973252313 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f318002a080 con 0x7f3194097360 2023-09-30T01:58:51.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.771+0000 7f3190ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:51.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.771+0000 7f3190ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:58:51.771 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:58:51.772 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:51.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.773+0000 7f319b21a700 1 -- 172.21.15.7:0/3973252313 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3178035810 msgr2=0x7f3178037cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:51.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.773+0000 7f319b21a700 1 --2- 172.21.15.7:0/3973252313 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3178035810 0x7f3178037cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3188009eb0 tx=0x7f31880091a0 comp rx=0 tx=0).stop 2023-09-30T01:58:51.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.773+0000 7f319b21a700 10 monclient: shutdown 2023-09-30T01:58:51.773 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.773+0000 7f319b21a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:51.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.773+0000 7f319b21a700 1 -- 172.21.15.7:0/3973252313 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3194097360 msgr2=0x7f319409db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:51.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.773+0000 7f319b21a700 1 --2- 172.21.15.7:0/3973252313 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3194097360 0x7f319409db90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3180006fd0 tx=0x7f3180017000 comp rx=0 tx=0).stop 2023-09-30T01:58:51.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.773+0000 7f319b21a700 1 -- 172.21.15.7:0/3973252313 shutdown_connections 2023-09-30T01:58:51.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.773+0000 7f319b21a700 1 --2- 172.21.15.7:0/3973252313 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3178035810 0x7f3178037cc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:51.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.773+0000 7f319b21a700 1 --2- 172.21.15.7:0/3973252313 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f31940a2390 0x7f319409e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:51.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.773+0000 7f319b21a700 1 --2- 172.21.15.7:0/3973252313 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3194096a50 0x7f319412df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:51.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.773+0000 7f319b21a700 1 --2- 172.21.15.7:0/3973252313 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3194097360 0x7f319409db90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:51.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.773+0000 7f319b21a700 1 -- 172.21.15.7:0/3973252313 >> 172.21.15.7:0/3973252313 conn(0x7f3194070cf0 msgr2=0x7f3194135780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:51.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.773+0000 7f319b21a700 1 -- 172.21.15.7:0/3973252313 shutdown_connections 2023-09-30T01:58:51.774 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:51.773+0000 7f319b21a700 1 -- 172.21.15.7:0/3973252313 wait complete. 2023-09-30T01:58:54.786 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:54.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.915+0000 7f09b2252700 10 monclient: get_monmap_and_config 2023-09-30T01:58:54.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.915+0000 7f09b2252700 10 monclient: build_initial_monmap 2023-09-30T01:58:54.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.915+0000 7f09b2252700 10 monclient: monmap: 2023-09-30T01:58:54.915 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:54.915 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:54.915 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:58:54.916289+0000 2023-09-30T01:58:54.915 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:58:54.916289+0000 2023-09-30T01:58:54.916 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:54.916 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:54.916 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:58:54.916 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:58:54.916 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:58:54.916 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:58:54.916 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:58:54.916 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:54.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09b2252700 1 Processor -- start 2023-09-30T01:58:54.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09b2252700 1 -- start start 2023-09-30T01:58:54.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09b2252700 10 monclient: init 2023-09-30T01:58:54.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09b2252700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:54.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09b2252700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-09-30T01:58:54.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09b2252700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f09ac096a50 0x7f09ac1327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:54.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09b2252700 10 monclient(hunting): picked mon.noname-c con 0x7f09ac096a50 addr v2:172.21.15.50:3300/0 2023-09-30T01:58:54.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09b2252700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09ac0a2390 addr v1:172.21.15.50:6789/0 2023-09-30T01:58:54.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09b2252700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09ac097530 0x7f09ac139720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:54.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09b2252700 10 monclient(hunting): picked mon.noname-b con 0x7f09ac097530 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:54.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09b2252700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ac0980e0 con 0x7f09ac0a2390 2023-09-30T01:58:54.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09b2252700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:54.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09b2252700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ac0991a0 con 0x7f09ac096a50 2023-09-30T01:58:54.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09b2252700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:54.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09b2252700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac099480 con 0x7f09ac097530 2023-09-30T01:58:54.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09b2252700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:54.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09b2252700 10 monclient(hunting): authenticate will time out at 1423.981607s 2023-09-30T01:58:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09abfff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09ac097530 0x7f09ac139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09abfff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09ac097530 0x7f09ac139720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:36168/0 (socket says 172.21.15.7:36168) 2023-09-30T01:58:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09abfff700 1 -- 172.21.15.7:0/1433283910 learned_addr learned my addr 172.21.15.7:0/1433283910 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09abfff700 10 monclient(hunting): get_auth_request con 0x7f09ac097530 auth_method 0 2023-09-30T01:58:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09abfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09abfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09abfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09abfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09abfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09ab7fe700 1 --2- 172.21.15.7:0/1433283910 >> v2:172.21.15.50:3300/0 conn(0x7f09ac096a50 0x7f09ac1327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09ab7fe700 10 monclient(hunting): get_auth_request con 0x7f09ac096a50 auth_method 0 2023-09-30T01:58:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09ab7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.916+0000 7f09ab7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.917+0000 7f09ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.917+0000 7f09ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.917+0000 7f09ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.917+0000 7f09abfff700 10 monclient(hunting): handle_auth_done global_id 4664 payload 274 2023-09-30T01:58:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.917+0000 7f09abfff700 1 -- 172.21.15.7:0/1433283910 >> v2:172.21.15.50:3300/0 conn(0x7f09ac096a50 msgr2=0x7f09ac1327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.917+0000 7f09abfff700 1 --2- 172.21.15.7:0/1433283910 >> v2:172.21.15.50:3300/0 conn(0x7f09ac096a50 0x7f09ac1327a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.917+0000 7f09abfff700 1 -- 172.21.15.7:0/1433283910 >> v1:172.21.15.50:6789/0 conn(0x7f09ac0a2390 legacy=0x7f09ac096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.917+0000 7f09abfff700 10 monclient: _finish_hunting 0 2023-09-30T01:58:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.917+0000 7f09abfff700 1 monclient: found mon.noname-b 2023-09-30T01:58:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.917+0000 7f09abfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.917+0000 7f09abfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:58:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.917+0000 7f09abfff700 1 -- 172.21.15.7:0/1433283910 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac097bf0 con 0x7f09ac097530 2023-09-30T01:58:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.919+0000 7f09abfff700 1 -- 172.21.15.7:0/1433283910 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09ac097530 msgr2=0x7f09ac139720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.919+0000 7f09abfff700 1 --2- 172.21.15.7:0/1433283910 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09ac097530 0x7f09ac139720 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f09a4009f20 tx=0x7f09a4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7524231c5a6b7c24 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.919+0000 7f09aa7fc700 1 -- 172.21.15.7:0/1433283910 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a400e070 con 0x7f09ac097530 2023-09-30T01:58:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.919+0000 7f09aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.919+0000 7f09aa7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:58:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.919+0000 7f09aa7fc700 10 monclient: dump: 2023-09-30T01:58:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:54.928 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.919+0000 7f09aa7fc700 10 monclient: _finish_auth 0 2023-09-30T01:58:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.919+0000 7f09aa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.919+0000 7f09aa7fc700 1 -- 172.21.15.7:0/1433283910 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a400bba0 con 0x7f09ac097530 2023-09-30T01:58:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.919+0000 7f09b2252700 5 monclient: authenticate success, global_id 4664 2023-09-30T01:58:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.919+0000 7f09aa7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.919+0000 7f09b2252700 10 monclient: get_monmap_and_config success 2023-09-30T01:58:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.919+0000 7f09aa7fc700 1 -- 172.21.15.7:0/1433283910 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a40092e0 con 0x7f09ac097530 2023-09-30T01:58:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.919+0000 7f09aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.919+0000 7f09b2252700 4 set_mon_vals no callback set 2023-09-30T01:58:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.919+0000 7f09aa7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:58:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.919+0000 7f09aa7fc700 10 monclient: dump: 2023-09-30T01:58:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:54.930 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:54.930 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:54.930 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:54.930 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:54.930 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:54.930 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:54.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.920+0000 7f09b2252700 10 monclient: shutdown 2023-09-30T01:58:54.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.920+0000 7f09b2252700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:54.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.920+0000 7f09b2252700 1 -- 172.21.15.7:0/1433283910 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09ac097530 msgr2=0x7f09ac139720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:54.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.920+0000 7f09b2252700 1 --2- 172.21.15.7:0/1433283910 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09ac097530 0x7f09ac139720 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f09a4009f20 tx=0x7f09a4009840 comp rx=0 tx=0).stop 2023-09-30T01:58:54.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.920+0000 7f09b2252700 1 -- 172.21.15.7:0/1433283910 shutdown_connections 2023-09-30T01:58:54.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.920+0000 7f09b2252700 1 --2- 172.21.15.7:0/1433283910 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09ac097530 0x7f09ac139720 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.920+0000 7f09b2252700 1 --2- 172.21.15.7:0/1433283910 >> v2:172.21.15.50:3300/0 conn(0x7f09ac096a50 0x7f09ac1327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.920+0000 7f09b2252700 1 -- 172.21.15.7:0/1433283910 >> 172.21.15.7:0/1433283910 conn(0x7f09ac070cf0 msgr2=0x7f09ac0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.920+0000 7f09b2252700 1 -- 172.21.15.7:0/1433283910 shutdown_connections 2023-09-30T01:58:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.920+0000 7f09b2252700 1 -- 172.21.15.7:0/1433283910 wait complete. 2023-09-30T01:58:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.920+0000 7f09b2252700 10 monclient: build_initial_monmap 2023-09-30T01:58:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.920+0000 7f09b2252700 10 monclient: monmap: 2023-09-30T01:58:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:58:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:58:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:58:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:54.932 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:58:54.932 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:58:54.932 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:58:54.933 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:54.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09b2252700 1 Processor -- start 2023-09-30T01:58:54.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09b2252700 1 -- start start 2023-09-30T01:58:54.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09b2252700 10 monclient: init 2023-09-30T01:58:54.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09b2252700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:54.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09b2252700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:58:54.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09b2252700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09ac096a50 0x7f09ac09b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:54.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09b2252700 10 monclient(hunting): picked mon.noname-b con 0x7f09ac096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:58:54.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09b2252700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09ac097530 0x7f09ac09bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:54.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09b2252700 10 monclient(hunting): picked mon.noname-a con 0x7f09ac097530 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:58:54.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09b2252700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09ac0a2390 0x7f09ac09c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:54.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09b2252700 10 monclient(hunting): picked mon.noname-c con 0x7f09ac0a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:54.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09b2252700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:54.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09b2252700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac1334e0 con 0x7f09ac097530 2023-09-30T01:58:54.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09b2252700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:54.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09b2252700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac1303c0 con 0x7f09ac096a50 2023-09-30T01:58:54.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09b2252700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:54.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09b2252700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac09f2c0 con 0x7f09ac0a2390 2023-09-30T01:58:54.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09b2252700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:54.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09b2252700 10 monclient(hunting): authenticate will time out at 1423.986672s 2023-09-30T01:58:54.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09aaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09ac097530 0x7f09ac09bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:54.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09abfff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09ac0a2390 0x7f09ac09c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:54.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09abfff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09ac0a2390 0x7f09ac09c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:36182/0 (socket says 172.21.15.7:36182) 2023-09-30T01:58:54.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09abfff700 1 -- 172.21.15.7:0/2316866592 learned_addr learned my addr 172.21.15.7:0/2316866592 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:54.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09abfff700 10 monclient(hunting): get_auth_request con 0x7f09ac0a2390 auth_method 0 2023-09-30T01:58:54.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09aaffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09ac097530 0x7f09ac09bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37424/0 (socket says 172.21.15.7:37424) 2023-09-30T01:58:54.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:54.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09abfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:54.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09abfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:54.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09aaffd700 10 monclient(hunting): get_auth_request con 0x7f09ac097530 auth_method 0 2023-09-30T01:58:54.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09aaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:54.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09aaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:54.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09aaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:54.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09abfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:54.935 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09abfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:54.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09abfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:54.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09aaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:54.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09aaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:54.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09aaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:54.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.921+0000 7f09ab7fe700 1 --2- 172.21.15.7:0/2316866592 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09ac096a50 0x7f09ac09b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:54.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.922+0000 7f09ab7fe700 10 monclient(hunting): get_auth_request con 0x7f09ac096a50 auth_method 0 2023-09-30T01:58:54.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.922+0000 7f09ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:54.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.922+0000 7f09ab7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:54.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.922+0000 7f09ab7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:54.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.922+0000 7f09abfff700 10 monclient(hunting): handle_auth_done global_id 4667 payload 824 2023-09-30T01:58:54.936 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.922+0000 7f09abfff700 1 -- 172.21.15.7:0/2316866592 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09ac096a50 msgr2=0x7f09ac09b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:54.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.922+0000 7f09abfff700 1 --2- 172.21.15.7:0/2316866592 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09ac096a50 0x7f09ac09b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:54.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.922+0000 7f09abfff700 1 -- 172.21.15.7:0/2316866592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09ac097530 msgr2=0x7f09ac09bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:54.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.922+0000 7f09abfff700 1 --2- 172.21.15.7:0/2316866592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09ac097530 0x7f09ac09bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:54.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.922+0000 7f09abfff700 10 monclient: _finish_hunting 0 2023-09-30T01:58:54.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.922+0000 7f09abfff700 1 monclient: found mon.noname-c 2023-09-30T01:58:54.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.922+0000 7f09abfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:54.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.922+0000 7f09abfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:58:54.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.922+0000 7f09abfff700 1 -- 172.21.15.7:0/2316866592 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a4009ae0 con 0x7f09ac0a2390 2023-09-30T01:58:54.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.922+0000 7f09aaffd700 1 --2- 172.21.15.7:0/2316866592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09ac097530 0x7f09ac09bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:58:54.937 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.922+0000 7f09ab7fe700 1 --2- 172.21.15.7:0/2316866592 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09ac096a50 0x7f09ac09b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:58:54.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09abfff700 1 -- 172.21.15.7:0/2316866592 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09ac0a2390 msgr2=0x7f09ac09c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:54.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09abfff700 1 --2- 172.21.15.7:0/2316866592 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09ac0a2390 0x7f09ac09c300 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f09ac0995c0 tx=0x7f09a4017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:54.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09937fe700 1 -- 172.21.15.7:0/2316866592 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a4012b60 con 0x7f09ac0a2390 2023-09-30T01:58:54.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:54.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09937fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:58:54.940 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09937fe700 10 monclient: dump: 2023-09-30T01:58:54.940 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:54.940 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:54.941 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:54.941 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:54.941 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:54.941 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:54.941 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:54.941 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:54.941 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:54.941 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:54.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09937fe700 10 monclient: _finish_auth 0 2023-09-30T01:58:54.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09937fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:54.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09b2252700 5 monclient: authenticate success, global_id 4667 2023-09-30T01:58:54.941 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09937fe700 1 -- 172.21.15.7:0/2316866592 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a4012cc0 con 0x7f09ac0a2390 2023-09-30T01:58:54.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09b2252700 10 monclient: _renew_subs 2023-09-30T01:58:54.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09b2252700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:54.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09b2252700 1 -- 172.21.15.7:0/2316866592 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09ac09ecc0 con 0x7f09ac0a2390 2023-09-30T01:58:54.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09937fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:54.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09937fe700 1 -- 172.21.15.7:0/2316866592 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a401d670 con 0x7f09ac0a2390 2023-09-30T01:58:54.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:54.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09937fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:58:54.942 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09937fe700 10 monclient: dump: 2023-09-30T01:58:54.942 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:54.942 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:54.942 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:54.943 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:54.943 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:54.943 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:54.943 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:54.943 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:54.943 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:54.943 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:54.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09a9ffb700 4 set_mon_vals no callback set 2023-09-30T01:58:54.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09b2252700 10 monclient: _renew_subs 2023-09-30T01:58:54.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09b2252700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:54.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.924+0000 7f09b2252700 1 -- 172.21.15.7:0/2316866592 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ac09d4b0 con 0x7f09ac0a2390 2023-09-30T01:58:54.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.925+0000 7f09937fe700 1 -- 172.21.15.7:0/2316866592 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f09a401daa0 con 0x7f09ac0a2390 2023-09-30T01:58:54.943 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.925+0000 7f09b2252700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:54.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.925+0000 7f09b2252700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:54.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.925+0000 7f09b2252700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:54.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.925+0000 7f09b2252700 1 -- 172.21.15.7:0/2316866592 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09ac071150 con 0x7f09ac0a2390 2023-09-30T01:58:54.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.925+0000 7f09937fe700 1 --2- 172.21.15.7:0/2316866592 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f0998035830 0x7f0998037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:54.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.925+0000 7f09937fe700 1 -- 172.21.15.7:0/2316866592 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f09a40493c0 con 0x7f09ac0a2390 2023-09-30T01:58:54.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.926+0000 7f09ab7fe700 1 --2- 172.21.15.7:0/2316866592 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f0998035830 0x7f0998037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:54.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.926+0000 7f09ab7fe700 10 monclient: get_auth_request con 0x7f0998035830 auth_method 0 2023-09-30T01:58:54.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.928+0000 7f09ab7fe700 1 --2- 172.21.15.7:0/2316866592 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f0998035830 0x7f0998037ce0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f099c009d80 tx=0x7f099c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:54.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.930+0000 7f09937fe700 1 -- 172.21.15.7:0/2316866592 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f09a4012e30 con 0x7f09ac0a2390 2023-09-30T01:58:54.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.930+0000 7f09937fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:54.944 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:54.930+0000 7f09937fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:58:55.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:55.168+0000 7f09b2252700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:58:55.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:55.168+0000 7f09b2252700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:55.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:55.168+0000 7f09b2252700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:58:55.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:55.168+0000 7f09b2252700 1 -- 172.21.15.7:0/2316866592 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f09ac064c10 con 0x7f09ac0a2390 2023-09-30T01:58:55.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:55.168+0000 7f09937fe700 1 -- 172.21.15.7:0/2316866592 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f09a4009910 con 0x7f09ac0a2390 2023-09-30T01:58:55.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:55.168+0000 7f09937fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:55.169 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:55.168+0000 7f09937fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:58:55.169 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:58:55.169 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:55.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:55.170+0000 7f09b2252700 1 -- 172.21.15.7:0/2316866592 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f0998035830 msgr2=0x7f0998037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:55.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:55.170+0000 7f09b2252700 1 --2- 172.21.15.7:0/2316866592 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f0998035830 0x7f0998037ce0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f099c009d80 tx=0x7f099c008040 comp rx=0 tx=0).stop 2023-09-30T01:58:55.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:55.170+0000 7f09b2252700 10 monclient: shutdown 2023-09-30T01:58:55.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:55.170+0000 7f09b2252700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:55.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:55.170+0000 7f09b2252700 1 -- 172.21.15.7:0/2316866592 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09ac0a2390 msgr2=0x7f09ac09c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:55.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:55.170+0000 7f09b2252700 1 --2- 172.21.15.7:0/2316866592 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09ac0a2390 0x7f09ac09c300 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f09ac0995c0 tx=0x7f09a4017000 comp rx=0 tx=0).stop 2023-09-30T01:58:55.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:55.170+0000 7f09b2252700 1 -- 172.21.15.7:0/2316866592 shutdown_connections 2023-09-30T01:58:55.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:55.170+0000 7f09b2252700 1 --2- 172.21.15.7:0/2316866592 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f0998035830 0x7f0998037ce0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:55.171 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:55.170+0000 7f09b2252700 1 --2- 172.21.15.7:0/2316866592 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f09ac0a2390 0x7f09ac09c300 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:55.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:55.170+0000 7f09b2252700 1 --2- 172.21.15.7:0/2316866592 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09ac096a50 0x7f09ac09b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:55.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:55.170+0000 7f09b2252700 1 --2- 172.21.15.7:0/2316866592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09ac097530 0x7f09ac09bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:55.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:55.170+0000 7f09b2252700 1 -- 172.21.15.7:0/2316866592 >> 172.21.15.7:0/2316866592 conn(0x7f09ac070cf0 msgr2=0x7f09ac1313e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:55.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:55.170+0000 7f09b2252700 1 -- 172.21.15.7:0/2316866592 shutdown_connections 2023-09-30T01:58:55.172 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:55.170+0000 7f09b2252700 1 -- 172.21.15.7:0/2316866592 wait complete. 2023-09-30T01:58:58.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:58:58.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.313+0000 7f3f87588700 10 monclient: get_monmap_and_config 2023-09-30T01:58:58.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.313+0000 7f3f87588700 10 monclient: build_initial_monmap 2023-09-30T01:58:58.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.313+0000 7f3f87588700 10 monclient: monmap: 2023-09-30T01:58:58.314 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:58.314 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:58.314 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:58:58.314666+0000 2023-09-30T01:58:58.314 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:58:58.314666+0000 2023-09-30T01:58:58.314 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:58.314 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:58.314 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:58:58.315 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:58:58.315 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:58:58.315 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:58:58.315 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:58:58.315 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:58.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.314+0000 7f3f87588700 1 Processor -- start 2023-09-30T01:58:58.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.314+0000 7f3f87588700 1 -- start start 2023-09-30T01:58:58.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.314+0000 7f3f87588700 10 monclient: init 2023-09-30T01:58:58.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.314+0000 7f3f87588700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:58.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.314+0000 7f3f87588700 10 monclient: _add_conns ranks=[1,3,4,2,0] 2023-09-30T01:58:58.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.314+0000 7f3f87588700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f80096a50 addr v1:172.21.15.50:6789/0 2023-09-30T01:58:58.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.314+0000 7f3f87588700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3f800a2390 0x7f3f80096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:58.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.314+0000 7f3f87588700 10 monclient(hunting): picked mon.noname-a con 0x7f3f800a2390 addr v2:172.21.15.7:3300/0 2023-09-30T01:58:58.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.314+0000 7f3f87588700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3f80097360 0x7f3f80139720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:58.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.314+0000 7f3f87588700 10 monclient(hunting): picked mon.noname-c con 0x7f3f80097360 addr v2:172.21.15.50:3300/0 2023-09-30T01:58:58.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.314+0000 7f3f87588700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f800980e0 con 0x7f3f80096a50 2023-09-30T01:58:58.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.314+0000 7f3f87588700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:58.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.314+0000 7f3f87588700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f80132eb0 con 0x7f3f800a2390 2023-09-30T01:58:58.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.314+0000 7f3f87588700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:58.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.314+0000 7f3f87588700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f800991a0 con 0x7f3f80097360 2023-09-30T01:58:58.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.314+0000 7f3f87588700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:58.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.314+0000 7f3f87588700 10 monclient(hunting): authenticate will time out at 1427.379982s 2023-09-30T01:58:58.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f84b23700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3f800a2390 0x7f3f80096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:58.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f84b23700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3f800a2390 0x7f3f80096e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37440/0 (socket says 172.21.15.7:37440) 2023-09-30T01:58:58.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f84b23700 1 -- 172.21.15.7:0/289256873 learned_addr learned my addr 172.21.15.7:0/289256873 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:58.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f84b23700 10 monclient(hunting): get_auth_request con 0x7f3f800a2390 auth_method 0 2023-09-30T01:58:58.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f84b23700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:58.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f84b23700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:58.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f84b23700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:58.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f84b23700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:58.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f84b23700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:58.327 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f84b23700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:58.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f85b25700 1 --2- 172.21.15.7:0/289256873 >> v2:172.21.15.50:3300/0 conn(0x7f3f80097360 0x7f3f80139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:58.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f85b25700 10 monclient(hunting): get_auth_request con 0x7f3f80097360 auth_method 0 2023-09-30T01:58:58.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f85b25700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:58.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f85b25700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:58.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f85b25700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:58.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f84b23700 10 monclient(hunting): handle_auth_done global_id 4650 payload 274 2023-09-30T01:58:58.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f84b23700 1 -- 172.21.15.7:0/289256873 >> v2:172.21.15.50:3300/0 conn(0x7f3f80097360 msgr2=0x7f3f80139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:58.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f84b23700 1 --2- 172.21.15.7:0/289256873 >> v2:172.21.15.50:3300/0 conn(0x7f3f80097360 0x7f3f80139720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:58.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f84b23700 1 -- 172.21.15.7:0/289256873 >> v1:172.21.15.50:6789/0 conn(0x7f3f80096a50 legacy=0x7f3f801327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:58:58.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f84b23700 10 monclient: _finish_hunting 0 2023-09-30T01:58:58.328 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f84b23700 1 monclient: found mon.noname-a 2023-09-30T01:58:58.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f84b23700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:58.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f84b23700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:58:58.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.315+0000 7f3f84b23700 1 -- 172.21.15.7:0/289256873 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f80099480 con 0x7f3f800a2390 2023-09-30T01:58:58.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.317+0000 7f3f84b23700 1 -- 172.21.15.7:0/289256873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f800a2390 msgr2=0x7f3f80096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:58.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.317+0000 7f3f84b23700 1 --2- 172.21.15.7:0/289256873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f800a2390 0x7f3f80096e20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3f80098850 tx=0x7f3f74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1cded49a5f5e6e8 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:58.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f7ffff700 1 -- 172.21.15.7:0/289256873 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7400ba30 con 0x7f3f800a2390 2023-09-30T01:58:58.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f7ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:58.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f7ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:58:58.329 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f7ffff700 10 monclient: dump: 2023-09-30T01:58:58.329 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:58.330 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:58.330 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:58.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f7ffff700 10 monclient: _finish_auth 0 2023-09-30T01:58:58.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f7ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:58.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f7ffff700 1 -- 172.21.15.7:0/289256873 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7400bb90 con 0x7f3f800a2390 2023-09-30T01:58:58.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f87588700 5 monclient: authenticate success, global_id 4650 2023-09-30T01:58:58.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f87588700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:58:58.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f7ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:58.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f87588700 10 monclient: get_monmap_and_config success 2023-09-30T01:58:58.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f7ffff700 1 -- 172.21.15.7:0/289256873 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f74011420 con 0x7f3f800a2390 2023-09-30T01:58:58.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f87588700 4 set_mon_vals no callback set 2023-09-30T01:58:58.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f7ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:58.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f7ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:58:58.331 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f7ffff700 10 monclient: dump: 2023-09-30T01:58:58.332 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:58.332 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:58.332 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:58.332 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:58.332 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:58.332 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:58.332 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:58.332 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:58.332 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:58.332 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:58.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f87588700 10 monclient: shutdown 2023-09-30T01:58:58.332 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f87588700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:58.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f87588700 1 -- 172.21.15.7:0/289256873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f800a2390 msgr2=0x7f3f80096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:58.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f87588700 1 --2- 172.21.15.7:0/289256873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f800a2390 0x7f3f80096e20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3f80098850 tx=0x7f3f74009250 comp rx=0 tx=0).stop 2023-09-30T01:58:58.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f87588700 1 -- 172.21.15.7:0/289256873 shutdown_connections 2023-09-30T01:58:58.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f87588700 1 --2- 172.21.15.7:0/289256873 >> v2:172.21.15.50:3300/0 conn(0x7f3f80097360 0x7f3f80139720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:58.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f87588700 1 --2- 172.21.15.7:0/289256873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f800a2390 0x7f3f80096e20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:58.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f87588700 1 -- 172.21.15.7:0/289256873 >> 172.21.15.7:0/289256873 conn(0x7f3f80070cf0 msgr2=0x7f3f800710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:58.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f87588700 1 -- 172.21.15.7:0/289256873 shutdown_connections 2023-09-30T01:58:58.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.318+0000 7f3f87588700 1 -- 172.21.15.7:0/289256873 wait complete. 2023-09-30T01:58:58.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.319+0000 7f3f87588700 10 monclient: build_initial_monmap 2023-09-30T01:58:58.333 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.319+0000 7f3f87588700 10 monclient: monmap: 2023-09-30T01:58:58.333 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:58:58.334 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:58:58.334 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:58:58.334 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:58:58.334 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:58:58.334 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:58.334 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:58:58.334 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:58:58.334 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:58:58.334 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:58.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.319+0000 7f3f87588700 1 Processor -- start 2023-09-30T01:58:58.334 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.319+0000 7f3f87588700 1 -- start start 2023-09-30T01:58:58.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.319+0000 7f3f87588700 10 monclient: init 2023-09-30T01:58:58.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f87588700 10 monclient: _reopen_session rank -1 2023-09-30T01:58:58.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f87588700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:58:58.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f87588700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f80096a50 0x7f3f8012f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:58.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f87588700 10 monclient(hunting): picked mon.noname-a con 0x7f3f80096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:58:58.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f87588700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3f80097360 0x7f3f8012f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:58.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f87588700 10 monclient(hunting): picked mon.noname-c con 0x7f3f80097360 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:58:58.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f87588700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f800a2390 0x7f3f8009bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:58.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f87588700 10 monclient(hunting): picked mon.noname-b con 0x7f3f800a2390 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:58:58.335 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f87588700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:58.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f87588700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f80130120 con 0x7f3f80096a50 2023-09-30T01:58:58.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f87588700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:58.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f87588700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8009f810 con 0x7f3f800a2390 2023-09-30T01:58:58.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f87588700 10 monclient(hunting): start opening mon connection 2023-09-30T01:58:58.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f87588700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8009f530 con 0x7f3f80097360 2023-09-30T01:58:58.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f87588700 10 monclient(hunting): _renew_subs 2023-09-30T01:58:58.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f87588700 10 monclient(hunting): authenticate will time out at 1427.385325s 2023-09-30T01:58:58.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85324700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f80096a50 0x7f3f8012f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:58.336 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f84b23700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3f80097360 0x7f3f8012f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:58.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85324700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f80096a50 0x7f3f8012f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37446/0 (socket says 172.21.15.7:37446) 2023-09-30T01:58:58.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85324700 1 -- 172.21.15.7:0/2548334828 learned_addr learned my addr 172.21.15.7:0/2548334828 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:58:58.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85324700 10 monclient(hunting): get_auth_request con 0x7f3f80096a50 auth_method 0 2023-09-30T01:58:58.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85324700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:58.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85324700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:58.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85324700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:58.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f84b23700 10 monclient(hunting): get_auth_request con 0x7f3f80097360 auth_method 0 2023-09-30T01:58:58.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f84b23700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:58:58.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f84b23700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:58:58.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f84b23700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:58:58.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85324700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:58.337 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85324700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:58.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85324700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:58.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f84b23700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:58:58.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f84b23700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:58:58.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f84b23700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:58:58.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85b25700 1 --2- 172.21.15.7:0/2548334828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f800a2390 0x7f3f8009bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:58.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85324700 10 monclient(hunting): handle_auth_done global_id 4653 payload 824 2023-09-30T01:58:58.338 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85324700 1 -- 172.21.15.7:0/2548334828 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3f80097360 msgr2=0x7f3f8012f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:58.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85324700 1 --2- 172.21.15.7:0/2548334828 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3f80097360 0x7f3f8012f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:58.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85324700 1 -- 172.21.15.7:0/2548334828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f800a2390 msgr2=0x7f3f8009bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:58.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85324700 1 --2- 172.21.15.7:0/2548334828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f800a2390 0x7f3f8009bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:58.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85324700 10 monclient: _finish_hunting 0 2023-09-30T01:58:58.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85324700 1 monclient: found mon.noname-a 2023-09-30T01:58:58.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85324700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:58:58.339 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85324700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:58:58.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85324700 1 -- 172.21.15.7:0/2548334828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f740094f0 con 0x7f3f80096a50 2023-09-30T01:58:58.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85b25700 10 monclient: get_auth_request con 0x7f3f800a2390 auth_method 0 2023-09-30T01:58:58.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.320+0000 7f3f85b25700 1 --2- 172.21.15.7:0/2548334828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f800a2390 0x7f3f8009bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:58:58.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f85324700 1 -- 172.21.15.7:0/2548334828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f80096a50 msgr2=0x7f3f8012f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:58:58.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f85324700 1 --2- 172.21.15.7:0/2548334828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f80096a50 0x7f3f8012f150 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3f7800c950 tx=0x7f3f78002760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:58.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f7d7fa700 1 -- 172.21.15.7:0/2548334828 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f78012840 con 0x7f3f80096a50 2023-09-30T01:58:58.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f7d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:58.340 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f7d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:58:58.341 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f7d7fa700 10 monclient: dump: 2023-09-30T01:58:58.341 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:58.341 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:58.341 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:58.341 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:58.341 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:58.341 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:58.342 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:58.342 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:58.342 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:58.342 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:58.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f7d7fa700 10 monclient: _finish_auth 0 2023-09-30T01:58:58.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f7d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:58:58.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f7d7fa700 1 -- 172.21.15.7:0/2548334828 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f780129c0 con 0x7f3f80096a50 2023-09-30T01:58:58.342 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f87588700 5 monclient: authenticate success, global_id 4653 2023-09-30T01:58:58.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f87588700 10 monclient: _renew_subs 2023-09-30T01:58:58.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f87588700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:58.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f87588700 1 -- 172.21.15.7:0/2548334828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f8009f250 con 0x7f3f80096a50 2023-09-30T01:58:58.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f7d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:58:58.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f7d7fa700 1 -- 172.21.15.7:0/2548334828 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f78012d80 con 0x7f3f80096a50 2023-09-30T01:58:58.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f7d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:58:58.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f7d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:58:58.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f7d7fa700 10 monclient: dump: 2023-09-30T01:58:58.346 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:58:58.346 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:58:58.346 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:58.346 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:58:58.346 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:58:58.346 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:58:58.347 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:58:58.347 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:58:58.347 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:58:58.347 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:58:58.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f7f7fe700 4 set_mon_vals no callback set 2023-09-30T01:58:58.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f87588700 10 monclient: _renew_subs 2023-09-30T01:58:58.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f87588700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:58.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.323+0000 7f3f87588700 1 -- 172.21.15.7:0/2548334828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f8009d7a0 con 0x7f3f80096a50 2023-09-30T01:58:58.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.324+0000 7f3f7d7fa700 1 -- 172.21.15.7:0/2548334828 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f3f7800a630 con 0x7f3f80096a50 2023-09-30T01:58:58.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.324+0000 7f3f87588700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:58.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.324+0000 7f3f87588700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:58.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.324+0000 7f3f87588700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:58.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.324+0000 7f3f87588700 1 -- 172.21.15.7:0/2548334828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f8009dc30 con 0x7f3f80096a50 2023-09-30T01:58:58.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.324+0000 7f3f7d7fa700 1 --2- 172.21.15.7:0/2548334828 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3f6c035830 0x7f3f6c037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:58:58.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.324+0000 7f3f7d7fa700 1 -- 172.21.15.7:0/2548334828 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f3f78048110 con 0x7f3f80096a50 2023-09-30T01:58:58.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.324+0000 7f3f84b23700 1 --2- 172.21.15.7:0/2548334828 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3f6c035830 0x7f3f6c037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:58:58.348 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.324+0000 7f3f84b23700 10 monclient: get_auth_request con 0x7f3f6c035830 auth_method 0 2023-09-30T01:58:58.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.327+0000 7f3f84b23700 1 --2- 172.21.15.7:0/2548334828 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3f6c035830 0x7f3f6c037ce0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3f80098850 tx=0x7f3f74016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:58:58.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.329+0000 7f3f7d7fa700 1 -- 172.21.15.7:0/2548334828 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3f780228f0 con 0x7f3f80096a50 2023-09-30T01:58:58.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.329+0000 7f3f7d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:58:58.349 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.329+0000 7f3f7d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:58:58.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.568+0000 7f3f87588700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:58:58.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.568+0000 7f3f87588700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:58.569 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.568+0000 7f3f87588700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:58:58.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.568+0000 7f3f87588700 1 -- 172.21.15.7:0/2548334828 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3f800a4910 con 0x7f3f80096a50 2023-09-30T01:58:58.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.569+0000 7f3f7d7fa700 1 -- 172.21.15.7:0/2548334828 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3f78016080 con 0x7f3f80096a50 2023-09-30T01:58:58.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.569+0000 7f3f7d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:58:58.570 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.569+0000 7f3f7d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:58:58.571 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:58:58.571 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:58:58.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.570+0000 7f3f87588700 1 -- 172.21.15.7:0/2548334828 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3f6c035830 msgr2=0x7f3f6c037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:58.572 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.570+0000 7f3f87588700 1 --2- 172.21.15.7:0/2548334828 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3f6c035830 0x7f3f6c037ce0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3f80098850 tx=0x7f3f74016000 comp rx=0 tx=0).stop 2023-09-30T01:58:58.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.570+0000 7f3f87588700 10 monclient: shutdown 2023-09-30T01:58:58.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.570+0000 7f3f87588700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:58:58.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.570+0000 7f3f87588700 1 -- 172.21.15.7:0/2548334828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f80096a50 msgr2=0x7f3f8012f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:58:58.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.570+0000 7f3f87588700 1 --2- 172.21.15.7:0/2548334828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f80096a50 0x7f3f8012f150 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3f7800c950 tx=0x7f3f78002760 comp rx=0 tx=0).stop 2023-09-30T01:58:58.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.571+0000 7f3f87588700 1 -- 172.21.15.7:0/2548334828 shutdown_connections 2023-09-30T01:58:58.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.571+0000 7f3f87588700 1 --2- 172.21.15.7:0/2548334828 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3f6c035830 0x7f3f6c037ce0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:58.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.571+0000 7f3f87588700 1 --2- 172.21.15.7:0/2548334828 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3f80096a50 0x7f3f8012f150 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:58.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.571+0000 7f3f87588700 1 --2- 172.21.15.7:0/2548334828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f800a2390 0x7f3f8009bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:58.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.571+0000 7f3f87588700 1 --2- 172.21.15.7:0/2548334828 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3f80097360 0x7f3f8012f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:58:58.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.571+0000 7f3f87588700 1 -- 172.21.15.7:0/2548334828 >> 172.21.15.7:0/2548334828 conn(0x7f3f80070cf0 msgr2=0x7f3f80130770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:58:58.573 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.571+0000 7f3f87588700 1 -- 172.21.15.7:0/2548334828 shutdown_connections 2023-09-30T01:58:58.574 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:58:58.571+0000 7f3f87588700 1 -- 172.21.15.7:0/2548334828 wait complete. 2023-09-30T01:59:01.584 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:01.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.715+0000 7f41f48b4700 10 monclient: get_monmap_and_config 2023-09-30T01:59:01.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.715+0000 7f41f48b4700 10 monclient: build_initial_monmap 2023-09-30T01:59:01.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.715+0000 7f41f48b4700 10 monclient: monmap: 2023-09-30T01:59:01.716 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:01.716 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:01.716 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:01.716319+0000 2023-09-30T01:59:01.716 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:01.716319+0000 2023-09-30T01:59:01.716 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:01.716 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:01.716 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:01.716 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:01.716 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:01.717 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:01.717 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:01.717 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:01.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.715+0000 7f41f48b4700 1 Processor -- start 2023-09-30T01:59:01.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.715+0000 7f41f48b4700 1 -- start start 2023-09-30T01:59:01.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.715+0000 7f41f48b4700 10 monclient: init 2023-09-30T01:59:01.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41f48b4700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:01.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41f48b4700 10 monclient: _add_conns ranks=[4,2,1,0,3] 2023-09-30T01:59:01.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41f48b4700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f41f00a29e0 0x7f41f0096a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:01.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41f48b4700 10 monclient(hunting): picked mon.noname-c con 0x7f41f00a29e0 addr v2:172.21.15.50:3300/0 2023-09-30T01:59:01.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41f48b4700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41f0096f90 0x7f41f0097360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:01.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41f48b4700 10 monclient(hunting): picked mon.noname-b con 0x7f41f0096f90 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:01.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41f48b4700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f41f0132220 addr v1:172.21.15.50:6789/0 2023-09-30T01:59:01.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41f48b4700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f0069cc0 con 0x7f41f0132220 2023-09-30T01:59:01.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41f48b4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:01.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41f48b4700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41f0098ea0 con 0x7f41f00a29e0 2023-09-30T01:59:01.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41f48b4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:01.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41f48b4700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f00978a0 con 0x7f41f0096f90 2023-09-30T01:59:01.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41f48b4700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:01.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41f48b4700 10 monclient(hunting): authenticate will time out at 1430.781591s 2023-09-30T01:59:01.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41edd9b700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41f0096f90 0x7f41f0097360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:01.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41edd9b700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41f0096f90 0x7f41f0097360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:36192/0 (socket says 172.21.15.7:36192) 2023-09-30T01:59:01.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41edd9b700 1 -- 172.21.15.7:0/1799188465 learned_addr learned my addr 172.21.15.7:0/1799188465 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:01.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41edd9b700 10 monclient(hunting): get_auth_request con 0x7f41f0096f90 auth_method 0 2023-09-30T01:59:01.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41edd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:01.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41edd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:01.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41edd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:01.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41edd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:01.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41edd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:01.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41edd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:01.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41ee59c700 1 --2- 172.21.15.7:0/1799188465 >> v2:172.21.15.50:3300/0 conn(0x7f41f00a29e0 0x7f41f0096a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:01.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41ee59c700 10 monclient(hunting): get_auth_request con 0x7f41f00a29e0 auth_method 0 2023-09-30T01:59:01.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41ee59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:01.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41ee59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:01.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.716+0000 7f41ee59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:01.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.717+0000 7f41edd9b700 10 monclient(hunting): handle_auth_done global_id 4673 payload 274 2023-09-30T01:59:01.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.717+0000 7f41edd9b700 1 -- 172.21.15.7:0/1799188465 >> v2:172.21.15.50:3300/0 conn(0x7f41f00a29e0 msgr2=0x7f41f0096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:01.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.717+0000 7f41edd9b700 1 --2- 172.21.15.7:0/1799188465 >> v2:172.21.15.50:3300/0 conn(0x7f41f00a29e0 0x7f41f0096a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.717+0000 7f41edd9b700 1 -- 172.21.15.7:0/1799188465 >> v1:172.21.15.50:6789/0 conn(0x7f41f0132220 legacy=0x7f41f0132670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:01.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.717+0000 7f41edd9b700 10 monclient: _finish_hunting 0 2023-09-30T01:59:01.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.717+0000 7f41edd9b700 1 monclient: found mon.noname-b 2023-09-30T01:59:01.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.717+0000 7f41edd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:01.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.717+0000 7f41edd9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:01.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.717+0000 7f41edd9b700 1 -- 172.21.15.7:0/1799188465 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f00a2db0 con 0x7f41f0096f90 2023-09-30T01:59:01.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.717+0000 7f41ee59c700 1 --2- 172.21.15.7:0/1799188465 >> v2:172.21.15.50:3300/0 conn(0x7f41f00a29e0 0x7f41f0096a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:59:01.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.719+0000 7f41edd9b700 1 -- 172.21.15.7:0/1799188465 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41f0096f90 msgr2=0x7f41f0097360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:01.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.719+0000 7f41edd9b700 1 --2- 172.21.15.7:0/1799188465 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41f0096f90 0x7f41f0097360 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f41d80099b0 tx=0x7f41d8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fabda40705c4d25b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:01.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.719+0000 7f41ed59a700 1 -- 172.21.15.7:0/1799188465 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d800ba30 con 0x7f41f0096f90 2023-09-30T01:59:01.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.719+0000 7f41ed59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:01.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.719+0000 7f41ed59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:01.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.719+0000 7f41ed59a700 10 monclient: dump: 2023-09-30T01:59:01.727 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:01.727 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:01.728 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:01.728 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:01.728 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:01.728 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:01.728 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:01.728 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:01.728 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:01.728 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:01.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.719+0000 7f41ed59a700 10 monclient: _finish_auth 0 2023-09-30T01:59:01.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.719+0000 7f41ed59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:01.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.719+0000 7f41ed59a700 1 -- 172.21.15.7:0/1799188465 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d800bb90 con 0x7f41f0096f90 2023-09-30T01:59:01.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.719+0000 7f41f48b4700 5 monclient: authenticate success, global_id 4673 2023-09-30T01:59:01.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.719+0000 7f41f48b4700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:01.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.719+0000 7f41ed59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:01.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.719+0000 7f41f48b4700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:01.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.719+0000 7f41ed59a700 1 -- 172.21.15.7:0/1799188465 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d8011420 con 0x7f41f0096f90 2023-09-30T01:59:01.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.719+0000 7f41f48b4700 4 set_mon_vals no callback set 2023-09-30T01:59:01.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.719+0000 7f41ed59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:01.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.719+0000 7f41ed59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:01.729 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.719+0000 7f41ed59a700 10 monclient: dump: 2023-09-30T01:59:01.729 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:01.729 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:01.730 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:01.730 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:01.730 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:01.730 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:01.730 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:01.730 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:01.730 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:01.730 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:01.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.720+0000 7f41f48b4700 10 monclient: shutdown 2023-09-30T01:59:01.730 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.720+0000 7f41f48b4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:01.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.720+0000 7f41f48b4700 1 -- 172.21.15.7:0/1799188465 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41f0096f90 msgr2=0x7f41f0097360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:01.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.720+0000 7f41f48b4700 1 --2- 172.21.15.7:0/1799188465 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41f0096f90 0x7f41f0097360 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f41d80099b0 tx=0x7f41d8009250 comp rx=0 tx=0).stop 2023-09-30T01:59:01.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.720+0000 7f41f48b4700 1 -- 172.21.15.7:0/1799188465 shutdown_connections 2023-09-30T01:59:01.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.720+0000 7f41f48b4700 1 --2- 172.21.15.7:0/1799188465 >> v2:172.21.15.50:3300/0 conn(0x7f41f00a29e0 0x7f41f0096a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.720+0000 7f41f48b4700 1 --2- 172.21.15.7:0/1799188465 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41f0096f90 0x7f41f0097360 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.720+0000 7f41f48b4700 1 -- 172.21.15.7:0/1799188465 >> 172.21.15.7:0/1799188465 conn(0x7f41f00710d0 msgr2=0x7f41f00714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:01.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.720+0000 7f41f48b4700 1 -- 172.21.15.7:0/1799188465 shutdown_connections 2023-09-30T01:59:01.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.720+0000 7f41f48b4700 1 -- 172.21.15.7:0/1799188465 wait complete. 2023-09-30T01:59:01.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.720+0000 7f41f48b4700 10 monclient: build_initial_monmap 2023-09-30T01:59:01.731 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.720+0000 7f41f48b4700 10 monclient: monmap: 2023-09-30T01:59:01.731 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:01.732 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:01.732 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:01.732 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:01.732 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:01.732 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:01.732 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:01.732 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:01.732 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:01.732 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:01.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.720+0000 7f41f48b4700 1 Processor -- start 2023-09-30T01:59:01.732 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41f48b4700 1 -- start start 2023-09-30T01:59:01.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41f48b4700 10 monclient: init 2023-09-30T01:59:01.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41f48b4700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:01.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41f48b4700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:59:01.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41f48b4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41f0096f90 0x7f41f012d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:01.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41f48b4700 10 monclient(hunting): picked mon.noname-a con 0x7f41f0096f90 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:01.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41f48b4700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41f00a29e0 0x7f41f012dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:01.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41f48b4700 10 monclient(hunting): picked mon.noname-c con 0x7f41f00a29e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:01.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41f48b4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41f0132220 0x7f41f009cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:01.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41f48b4700 10 monclient(hunting): picked mon.noname-b con 0x7f41f0132220 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:01.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41f48b4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:01.733 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41f48b4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f00a01c0 con 0x7f41f0096f90 2023-09-30T01:59:01.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41f48b4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:01.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41f48b4700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f00a0810 con 0x7f41f0132220 2023-09-30T01:59:01.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41f48b4700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:01.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41f48b4700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f009e620 con 0x7f41f00a29e0 2023-09-30T01:59:01.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41f48b4700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:01.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41f48b4700 10 monclient(hunting): authenticate will time out at 1430.786498s 2023-09-30T01:59:01.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41edd9b700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41f00a29e0 0x7f41f012dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:01.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41ee59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41f0096f90 0x7f41f012d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:01.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41edd9b700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41f00a29e0 0x7f41f012dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:36204/0 (socket says 172.21.15.7:36204) 2023-09-30T01:59:01.734 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41edd9b700 1 -- 172.21.15.7:0/2689251532 learned_addr learned my addr 172.21.15.7:0/2689251532 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:01.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41edd9b700 10 monclient(hunting): get_auth_request con 0x7f41f00a29e0 auth_method 0 2023-09-30T01:59:01.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41edd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:01.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41edd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:01.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41edd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:01.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41ee59c700 10 monclient(hunting): get_auth_request con 0x7f41f0096f90 auth_method 0 2023-09-30T01:59:01.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41ee59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:01.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41ee59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:01.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41ee59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:01.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41edd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:01.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41edd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:01.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41edd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:01.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41ee59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:01.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41ee59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:01.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41ee59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:01.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41edd9b700 10 monclient(hunting): handle_auth_done global_id 4676 payload 824 2023-09-30T01:59:01.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41eed9d700 1 --2- 172.21.15.7:0/2689251532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41f0132220 0x7f41f009cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:01.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41edd9b700 1 -- 172.21.15.7:0/2689251532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41f0132220 msgr2=0x7f41f009cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:01.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41edd9b700 1 --2- 172.21.15.7:0/2689251532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41f0132220 0x7f41f009cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41edd9b700 1 -- 172.21.15.7:0/2689251532 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41f0096f90 msgr2=0x7f41f012d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:01.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41edd9b700 1 --2- 172.21.15.7:0/2689251532 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41f0096f90 0x7f41f012d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41edd9b700 10 monclient: _finish_hunting 0 2023-09-30T01:59:01.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41edd9b700 1 monclient: found mon.noname-c 2023-09-30T01:59:01.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41edd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:01.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41edd9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:59:01.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41edd9b700 1 -- 172.21.15.7:0/2689251532 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d80094f0 con 0x7f41f00a29e0 2023-09-30T01:59:01.737 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.721+0000 7f41ee59c700 1 --2- 172.21.15.7:0/2689251532 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41f0096f90 0x7f41f012d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:01.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41edd9b700 1 -- 172.21.15.7:0/2689251532 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41f00a29e0 msgr2=0x7f41f012dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:01.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41edd9b700 1 --2- 172.21.15.7:0/2689251532 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41f00a29e0 0x7f41f012dc60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f41d80090b0 tx=0x7f41d8016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:01.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41e67fc700 1 -- 172.21.15.7:0/2689251532 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d801a460 con 0x7f41f00a29e0 2023-09-30T01:59:01.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41e67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:01.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41e67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:59:01.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41e67fc700 10 monclient: dump: 2023-09-30T01:59:01.740 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:01.740 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:01.740 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:01.740 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:01.740 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:01.740 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:01.740 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:01.740 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:01.740 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:01.740 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:01.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41e67fc700 10 monclient: _finish_auth 0 2023-09-30T01:59:01.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41e67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:01.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41f48b4700 5 monclient: authenticate success, global_id 4676 2023-09-30T01:59:01.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41e67fc700 1 -- 172.21.15.7:0/2689251532 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d801a5c0 con 0x7f41f00a29e0 2023-09-30T01:59:01.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41e67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:01.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41e67fc700 1 -- 172.21.15.7:0/2689251532 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d801a980 con 0x7f41f00a29e0 2023-09-30T01:59:01.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41e67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:01.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41e67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:01.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41e67fc700 10 monclient: dump: 2023-09-30T01:59:01.741 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:01.741 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:01.742 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:01.742 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:01.742 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:01.742 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:01.742 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:01.742 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:01.742 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:01.742 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:01.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41ecd99700 4 set_mon_vals no callback set 2023-09-30T01:59:01.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41f48b4700 10 monclient: _renew_subs 2023-09-30T01:59:01.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41f48b4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:01.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41f48b4700 1 -- 172.21.15.7:0/2689251532 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f00a1220 con 0x7f41f00a29e0 2023-09-30T01:59:01.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41f48b4700 10 monclient: _renew_subs 2023-09-30T01:59:01.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41f48b4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:01.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.724+0000 7f41f48b4700 1 -- 172.21.15.7:0/2689251532 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f0131d50 con 0x7f41f00a29e0 2023-09-30T01:59:01.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.725+0000 7f41e67fc700 1 -- 172.21.15.7:0/2689251532 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f41d801b440 con 0x7f41f00a29e0 2023-09-30T01:59:01.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.725+0000 7f41f48b4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:01.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.725+0000 7f41f48b4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:01.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.725+0000 7f41f48b4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:01.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.725+0000 7f41f48b4700 1 -- 172.21.15.7:0/2689251532 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f0064c10 con 0x7f41f00a29e0 2023-09-30T01:59:01.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.725+0000 7f41e67fc700 1 --2- 172.21.15.7:0/2689251532 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f41d00357e0 0x7f41d0037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:01.743 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.725+0000 7f41e67fc700 1 -- 172.21.15.7:0/2689251532 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f41d8049440 con 0x7f41f00a29e0 2023-09-30T01:59:01.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.725+0000 7f41ee59c700 1 --2- 172.21.15.7:0/2689251532 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f41d00357e0 0x7f41d0037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:01.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.725+0000 7f41ee59c700 10 monclient: get_auth_request con 0x7f41d00357e0 auth_method 0 2023-09-30T01:59:01.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.728+0000 7f41ee59c700 1 --2- 172.21.15.7:0/2689251532 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f41d00357e0 0x7f41d0037c90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f41e0009fd0 tx=0x7f41e0009a20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:01.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.730+0000 7f41e67fc700 1 -- 172.21.15.7:0/2689251532 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f41d8009300 con 0x7f41f00a29e0 2023-09-30T01:59:01.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.730+0000 7f41e67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:01.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.730+0000 7f41e67fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.965+0000 7f41f48b4700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:59:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.965+0000 7f41f48b4700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.965+0000 7f41f48b4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.965+0000 7f41f48b4700 1 -- 172.21.15.7:0/2689251532 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f41f009ff00 con 0x7f41f00a29e0 2023-09-30T01:59:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.966+0000 7f41e67fc700 1 -- 172.21.15.7:0/2689251532 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f41d802a080 con 0x7f41f00a29e0 2023-09-30T01:59:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.966+0000 7f41e67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:01.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.966+0000 7f41e67fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:59:01.967 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:59:01.967 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.968+0000 7f41f48b4700 1 -- 172.21.15.7:0/2689251532 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f41d00357e0 msgr2=0x7f41d0037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.968+0000 7f41f48b4700 1 --2- 172.21.15.7:0/2689251532 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f41d00357e0 0x7f41d0037c90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f41e0009fd0 tx=0x7f41e0009a20 comp rx=0 tx=0).stop 2023-09-30T01:59:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.968+0000 7f41f48b4700 10 monclient: shutdown 2023-09-30T01:59:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.968+0000 7f41f48b4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.968+0000 7f41f48b4700 1 -- 172.21.15.7:0/2689251532 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41f00a29e0 msgr2=0x7f41f012dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.968+0000 7f41f48b4700 1 --2- 172.21.15.7:0/2689251532 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41f00a29e0 0x7f41f012dc60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f41d80090b0 tx=0x7f41d8016000 comp rx=0 tx=0).stop 2023-09-30T01:59:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.968+0000 7f41f48b4700 1 -- 172.21.15.7:0/2689251532 shutdown_connections 2023-09-30T01:59:01.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.968+0000 7f41f48b4700 1 --2- 172.21.15.7:0/2689251532 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f41d00357e0 0x7f41d0037c90 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.968+0000 7f41f48b4700 1 --2- 172.21.15.7:0/2689251532 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41f0096f90 0x7f41f012d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.968+0000 7f41f48b4700 1 --2- 172.21.15.7:0/2689251532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41f0132220 0x7f41f009cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.968+0000 7f41f48b4700 1 --2- 172.21.15.7:0/2689251532 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41f00a29e0 0x7f41f012dc60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:01.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.968+0000 7f41f48b4700 1 -- 172.21.15.7:0/2689251532 >> 172.21.15.7:0/2689251532 conn(0x7f41f00710d0 msgr2=0x7f41f00a3670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:01.970 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.968+0000 7f41f48b4700 1 -- 172.21.15.7:0/2689251532 shutdown_connections 2023-09-30T01:59:01.971 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:01.968+0000 7f41f48b4700 1 -- 172.21.15.7:0/2689251532 wait complete. 2023-09-30T01:59:04.981 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:05.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.111+0000 7feb3852a700 10 monclient: get_monmap_and_config 2023-09-30T01:59:05.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.111+0000 7feb3852a700 10 monclient: build_initial_monmap 2023-09-30T01:59:05.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.111+0000 7feb3852a700 10 monclient: monmap: 2023-09-30T01:59:05.111 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:05.111 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:05.111 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:05.112248+0000 2023-09-30T01:59:05.111 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:05.112248+0000 2023-09-30T01:59:05.112 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:05.112 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:05.112 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:05.112 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:05.112 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:05.112 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:05.112 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:05.112 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:05.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.111+0000 7feb3852a700 1 Processor -- start 2023-09-30T01:59:05.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.111+0000 7feb3852a700 1 -- start start 2023-09-30T01:59:05.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.111+0000 7feb3852a700 10 monclient: init 2023-09-30T01:59:05.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb3852a700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:05.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb3852a700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2023-09-30T01:59:05.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb3852a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feb30096a50 0x7feb301327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:05.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb3852a700 10 monclient(hunting): picked mon.noname-a con 0x7feb30096a50 addr v2:172.21.15.7:3300/0 2023-09-30T01:59:05.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb3852a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb300a2390 addr v1:172.21.15.7:6789/0 2023-09-30T01:59:05.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb3852a700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb30097530 0x7feb30139720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:05.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb3852a700 10 monclient(hunting): picked mon.noname-b con 0x7feb30097530 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:05.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb3852a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb300980e0 con 0x7feb300a2390 2023-09-30T01:59:05.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb3852a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:05.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb3852a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb300991a0 con 0x7feb30096a50 2023-09-30T01:59:05.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb3852a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:05.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb3852a700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb30099480 con 0x7feb30097530 2023-09-30T01:59:05.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb3852a700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:05.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb3852a700 10 monclient(hunting): authenticate will time out at 1434.177503s 2023-09-30T01:59:05.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb362c6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feb30096a50 0x7feb301327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:05.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb36ac7700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb30097530 0x7feb30139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:05.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb35ac5700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7feb300a2390 0x7feb30096e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42788/0 (socket says 172.21.15.7:42788) 2023-09-30T01:59:05.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb362c6700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7feb30096a50 0x7feb301327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38556/0 (socket says 172.21.15.7:38556) 2023-09-30T01:59:05.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb35ac5700 1 -- 172.21.15.7:0/3037500570 learned_addr learned my addr 172.21.15.7:0/3037500570 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:59:05.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb362c6700 10 monclient(hunting): get_auth_request con 0x7feb30096a50 auth_method 0 2023-09-30T01:59:05.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb362c6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:05.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb362c6700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:05.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb362c6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:05.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb36ac7700 10 monclient(hunting): get_auth_request con 0x7feb30097530 auth_method 0 2023-09-30T01:59:05.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb36ac7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:05.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb36ac7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:05.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb36ac7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:05.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb362c6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:05.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb362c6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:05.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb362c6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:05.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb36ac7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:05.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb36ac7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:05.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb36ac7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:05.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb36ac7700 10 monclient(hunting): handle_auth_done global_id 4679 payload 274 2023-09-30T01:59:05.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb36ac7700 1 -- 172.21.15.7:0/3037500570 >> v2:172.21.15.7:3300/0 conn(0x7feb30096a50 msgr2=0x7feb301327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:05.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb36ac7700 1 --2- 172.21.15.7:0/3037500570 >> v2:172.21.15.7:3300/0 conn(0x7feb30096a50 0x7feb301327a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:05.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb36ac7700 1 -- 172.21.15.7:0/3037500570 >> v1:172.21.15.7:6789/0 conn(0x7feb300a2390 legacy=0x7feb30096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:05.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb36ac7700 10 monclient: _finish_hunting 0 2023-09-30T01:59:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb36ac7700 1 monclient: found mon.noname-b 2023-09-30T01:59:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb36ac7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb36ac7700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb36ac7700 1 -- 172.21.15.7:0/3037500570 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb30097bf0 con 0x7feb30097530 2023-09-30T01:59:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.112+0000 7feb362c6700 1 --2- 172.21.15.7:0/3037500570 >> v2:172.21.15.7:3300/0 conn(0x7feb30096a50 0x7feb301327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb36ac7700 1 -- 172.21.15.7:0/3037500570 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb30097530 msgr2=0x7feb30139720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb36ac7700 1 --2- 172.21.15.7:0/3037500570 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb30097530 0x7feb30139720 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7feb28009e60 tx=0x7feb28009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb84a654957967b4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb352c4700 1 -- 172.21.15.7:0/3037500570 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2800e070 con 0x7feb30097530 2023-09-30T01:59:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb352c4700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb352c4700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:05.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb352c4700 10 monclient: dump: 2023-09-30T01:59:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:05.123 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb352c4700 10 monclient: _finish_auth 0 2023-09-30T01:59:05.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb352c4700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb352c4700 1 -- 172.21.15.7:0/3037500570 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2800bd30 con 0x7feb30097530 2023-09-30T01:59:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb3852a700 5 monclient: authenticate success, global_id 4679 2023-09-30T01:59:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb3852a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb352c4700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb3852a700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb3852a700 4 set_mon_vals no callback set 2023-09-30T01:59:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb352c4700 1 -- 172.21.15.7:0/3037500570 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2800bea0 con 0x7feb30097530 2023-09-30T01:59:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb352c4700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb352c4700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb352c4700 10 monclient: dump: 2023-09-30T01:59:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:05.124 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:05.125 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:05.125 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:05.125 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:05.125 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:05.125 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:05.125 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:05.125 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:05.125 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:05.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb3852a700 10 monclient: shutdown 2023-09-30T01:59:05.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb3852a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:05.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb3852a700 1 -- 172.21.15.7:0/3037500570 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb30097530 msgr2=0x7feb30139720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:05.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb3852a700 1 --2- 172.21.15.7:0/3037500570 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb30097530 0x7feb30139720 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7feb28009e60 tx=0x7feb28009780 comp rx=0 tx=0).stop 2023-09-30T01:59:05.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.115+0000 7feb3852a700 1 -- 172.21.15.7:0/3037500570 shutdown_connections 2023-09-30T01:59:05.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.116+0000 7feb3852a700 1 --2- 172.21.15.7:0/3037500570 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb30097530 0x7feb30139720 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:05.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.116+0000 7feb3852a700 1 --2- 172.21.15.7:0/3037500570 >> v2:172.21.15.7:3300/0 conn(0x7feb30096a50 0x7feb301327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:05.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.116+0000 7feb3852a700 1 -- 172.21.15.7:0/3037500570 >> 172.21.15.7:0/3037500570 conn(0x7feb30070cf0 msgr2=0x7feb300710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:05.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.116+0000 7feb3852a700 1 -- 172.21.15.7:0/3037500570 shutdown_connections 2023-09-30T01:59:05.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.116+0000 7feb3852a700 1 -- 172.21.15.7:0/3037500570 wait complete. 2023-09-30T01:59:05.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.116+0000 7feb3852a700 10 monclient: build_initial_monmap 2023-09-30T01:59:05.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.116+0000 7feb3852a700 10 monclient: monmap: 2023-09-30T01:59:05.127 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:05.127 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:05.127 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:05.127 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:05.128 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:05.128 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:05.128 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:05.128 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:05.128 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:05.128 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:05.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.116+0000 7feb3852a700 1 Processor -- start 2023-09-30T01:59:05.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.116+0000 7feb3852a700 1 -- start start 2023-09-30T01:59:05.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.116+0000 7feb3852a700 10 monclient: init 2023-09-30T01:59:05.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb3852a700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:05.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb3852a700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:59:05.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb3852a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb30096a50 0x7feb3009bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:05.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb3852a700 10 monclient(hunting): picked mon.noname-b con 0x7feb30096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:05.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb3852a700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb30097530 0x7feb3009c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:05.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb3852a700 10 monclient(hunting): picked mon.noname-c con 0x7feb30097530 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:05.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb3852a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb300a2390 0x7feb3009c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:05.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb3852a700 10 monclient(hunting): picked mon.noname-a con 0x7feb300a2390 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:05.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb3852a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:05.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb3852a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3009f780 con 0x7feb300a2390 2023-09-30T01:59:05.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb3852a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:05.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb3852a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb301334e0 con 0x7feb30096a50 2023-09-30T01:59:05.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb3852a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:05.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb3852a700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb300a03e0 con 0x7feb30097530 2023-09-30T01:59:05.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb3852a700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:05.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb3852a700 10 monclient(hunting): authenticate will time out at 1434.182351s 2023-09-30T01:59:05.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb35ac5700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb30097530 0x7feb3009c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:05.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb36ac7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb300a2390 0x7feb3009c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:05.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb35ac5700 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb30097530 msgr2=0x7feb3009c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-09-30T01:59:05.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb36ac7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb300a2390 0x7feb3009c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38562/0 (socket says 172.21.15.7:38562) 2023-09-30T01:59:05.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb35ac5700 1 -- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb30097530 msgr2=0x7feb3009c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-30T01:59:05.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb36ac7700 1 -- 172.21.15.7:0/3282143673 learned_addr learned my addr 172.21.15.7:0/3282143673 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:05.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb36ac7700 10 monclient(hunting): get_auth_request con 0x7feb300a2390 auth_method 0 2023-09-30T01:59:05.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb36ac7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:05.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb36ac7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:05.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb36ac7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:05.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb35ac5700 1 --2- 172.21.15.7:0/3282143673 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb30097530 0x7feb3009c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T01:59:05.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb35ac5700 1 --2- 172.21.15.7:0/3282143673 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb30097530 0x7feb3009c160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T01:59:05.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb36ac7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:05.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb36ac7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:05.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb36ac7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:05.131 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb36ac7700 10 monclient(hunting): handle_auth_done global_id 4662 payload 824 2023-09-30T01:59:05.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb36ac7700 1 -- 172.21.15.7:0/3282143673 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb30097530 msgr2=0x7feb3009c160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-30T01:59:05.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb362c6700 1 --2- 172.21.15.7:0/3282143673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb30096a50 0x7feb3009bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:05.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb36ac7700 1 --2- 172.21.15.7:0/3282143673 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb30097530 0x7feb3009c160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:05.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb36ac7700 1 -- 172.21.15.7:0/3282143673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb30096a50 msgr2=0x7feb3009bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:05.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb36ac7700 1 --2- 172.21.15.7:0/3282143673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb30096a50 0x7feb3009bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:05.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb36ac7700 10 monclient: _finish_hunting 0 2023-09-30T01:59:05.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb36ac7700 1 monclient: found mon.noname-a 2023-09-30T01:59:05.132 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb36ac7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:05.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb36ac7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:05.134 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.117+0000 7feb36ac7700 1 -- 172.21.15.7:0/3282143673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c009420 con 0x7feb300a2390 2023-09-30T01:59:05.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.119+0000 7feb36ac7700 1 -- 172.21.15.7:0/3282143673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb300a2390 msgr2=0x7feb3009c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:05.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.119+0000 7feb36ac7700 1 --2- 172.21.15.7:0/3282143673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb300a2390 0x7feb3009c6a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7feb2800b6c0 tx=0x7feb28018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:05.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.120+0000 7feb267fc700 1 -- 172.21.15.7:0/3282143673 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb28012e50 con 0x7feb300a2390 2023-09-30T01:59:05.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.120+0000 7feb267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:05.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.120+0000 7feb267fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:05.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.120+0000 7feb267fc700 10 monclient: dump: 2023-09-30T01:59:05.137 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:05.137 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:05.137 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:05.138 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:05.138 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:05.138 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:05.138 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:05.138 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:05.138 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:05.138 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:05.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.120+0000 7feb267fc700 10 monclient: _finish_auth 0 2023-09-30T01:59:05.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.120+0000 7feb267fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:05.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.120+0000 7feb267fc700 1 -- 172.21.15.7:0/3282143673 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb28021070 con 0x7feb300a2390 2023-09-30T01:59:05.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.120+0000 7feb3852a700 5 monclient: authenticate success, global_id 4662 2023-09-30T01:59:05.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.120+0000 7feb3852a700 10 monclient: _renew_subs 2023-09-30T01:59:05.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.120+0000 7feb3852a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:05.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.120+0000 7feb3852a700 1 -- 172.21.15.7:0/3282143673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb28009a20 con 0x7feb300a2390 2023-09-30T01:59:05.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.120+0000 7feb267fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:05.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.120+0000 7feb267fc700 1 -- 172.21.15.7:0/3282143673 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb28026660 con 0x7feb300a2390 2023-09-30T01:59:05.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.120+0000 7feb267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:05.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.120+0000 7feb267fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:05.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.120+0000 7feb267fc700 10 monclient: dump: 2023-09-30T01:59:05.139 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:05.140 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:05.140 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:05.140 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:05.140 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:05.140 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:05.140 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:05.140 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:05.140 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:05.140 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:05.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.120+0000 7feb3852a700 10 monclient: _renew_subs 2023-09-30T01:59:05.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.120+0000 7feb3852a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:05.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.120+0000 7feb3852a700 1 -- 172.21.15.7:0/3282143673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb3009d600 con 0x7feb300a2390 2023-09-30T01:59:05.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.120+0000 7feb34ac3700 4 set_mon_vals no callback set 2023-09-30T01:59:05.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.120+0000 7feb267fc700 1 -- 172.21.15.7:0/3282143673 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7feb28023030 con 0x7feb300a2390 2023-09-30T01:59:05.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.121+0000 7feb3852a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:05.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.121+0000 7feb3852a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:05.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.121+0000 7feb3852a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:05.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.121+0000 7feb3852a700 1 -- 172.21.15.7:0/3282143673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb3009f5e0 con 0x7feb300a2390 2023-09-30T01:59:05.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.121+0000 7feb267fc700 1 --2- 172.21.15.7:0/3282143673 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7feb140357e0 0x7feb14037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:05.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.121+0000 7feb267fc700 1 -- 172.21.15.7:0/3282143673 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7feb2804a3c0 con 0x7feb300a2390 2023-09-30T01:59:05.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.121+0000 7feb362c6700 1 --2- 172.21.15.7:0/3282143673 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7feb140357e0 0x7feb14037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:05.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.121+0000 7feb362c6700 10 monclient: get_auth_request con 0x7feb140357e0 auth_method 0 2023-09-30T01:59:05.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.124+0000 7feb362c6700 1 --2- 172.21.15.7:0/3282143673 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7feb140357e0 0x7feb14037c90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7feb2c006f00 tx=0x7feb2c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:05.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.125+0000 7feb267fc700 1 -- 172.21.15.7:0/3282143673 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feb3009f5e0 con 0x7feb300a2390 2023-09-30T01:59:05.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.125+0000 7feb267fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:05.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.125+0000 7feb267fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:05.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.364+0000 7feb3852a700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:59:05.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.364+0000 7feb3852a700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:05.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.364+0000 7feb3852a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:05.365 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.364+0000 7feb3852a700 1 -- 172.21.15.7:0/3282143673 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7feb3009f5e0 con 0x7feb300a2390 2023-09-30T01:59:05.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.365+0000 7feb267fc700 1 -- 172.21.15.7:0/3282143673 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7feb3009f5e0 con 0x7feb300a2390 2023-09-30T01:59:05.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.365+0000 7feb267fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:05.366 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.365+0000 7feb267fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:59:05.366 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:59:05.366 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:05.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.367+0000 7feb3852a700 1 -- 172.21.15.7:0/3282143673 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7feb140357e0 msgr2=0x7feb14037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:05.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.367+0000 7feb3852a700 1 --2- 172.21.15.7:0/3282143673 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7feb140357e0 0x7feb14037c90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7feb2c006f00 tx=0x7feb2c008040 comp rx=0 tx=0).stop 2023-09-30T01:59:05.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.367+0000 7feb3852a700 10 monclient: shutdown 2023-09-30T01:59:05.367 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.367+0000 7feb3852a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:05.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.367+0000 7feb3852a700 1 -- 172.21.15.7:0/3282143673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb300a2390 msgr2=0x7feb3009c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:05.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.367+0000 7feb3852a700 1 --2- 172.21.15.7:0/3282143673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb300a2390 0x7feb3009c6a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7feb2800b6c0 tx=0x7feb28018000 comp rx=0 tx=0).stop 2023-09-30T01:59:05.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.367+0000 7feb3852a700 1 -- 172.21.15.7:0/3282143673 shutdown_connections 2023-09-30T01:59:05.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.367+0000 7feb3852a700 1 --2- 172.21.15.7:0/3282143673 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7feb140357e0 0x7feb14037c90 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:05.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.367+0000 7feb3852a700 1 --2- 172.21.15.7:0/3282143673 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7feb300a2390 0x7feb3009c6a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:05.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.367+0000 7feb3852a700 1 --2- 172.21.15.7:0/3282143673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb30096a50 0x7feb3009bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:05.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.367+0000 7feb3852a700 1 --2- 172.21.15.7:0/3282143673 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7feb30097530 0x7feb3009c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:05.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.367+0000 7feb3852a700 1 -- 172.21.15.7:0/3282143673 >> 172.21.15.7:0/3282143673 conn(0x7feb30070cf0 msgr2=0x7feb30131380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:05.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.367+0000 7feb3852a700 1 -- 172.21.15.7:0/3282143673 shutdown_connections 2023-09-30T01:59:05.368 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:05.367+0000 7feb3852a700 1 -- 172.21.15.7:0/3282143673 wait complete. 2023-09-30T01:59:08.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:08.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f96466d5700 10 monclient: get_monmap_and_config 2023-09-30T01:59:08.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f96466d5700 10 monclient: build_initial_monmap 2023-09-30T01:59:08.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f96466d5700 10 monclient: monmap: 2023-09-30T01:59:08.509 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:08.509 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:08.509 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:08.510275+0000 2023-09-30T01:59:08.509 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:08.510275+0000 2023-09-30T01:59:08.510 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:08.510 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:08.510 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:08.510 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:08.510 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:08.510 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:08.510 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:08.510 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:08.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f96466d5700 1 Processor -- start 2023-09-30T01:59:08.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f96466d5700 1 -- start start 2023-09-30T01:59:08.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f96466d5700 10 monclient: init 2023-09-30T01:59:08.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f96466d5700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:08.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f96466d5700 10 monclient: _add_conns ranks=[0,1,3,2,4] 2023-09-30T01:59:08.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f96466d5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9640096a50 addr v1:172.21.15.7:6789/0 2023-09-30T01:59:08.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f96466d5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f96400a2390 addr v1:172.21.15.50:6789/0 2023-09-30T01:59:08.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f96466d5700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9640097530 0x7f964013a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:08.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f96466d5700 10 monclient(hunting): picked mon.noname-a con 0x7f9640097530 addr v2:172.21.15.7:3300/0 2023-09-30T01:59:08.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f96466d5700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96400980e0 con 0x7f9640096a50 2023-09-30T01:59:08.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f96466d5700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9640132eb0 con 0x7f96400a2390 2023-09-30T01:59:08.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f96466d5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:08.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f96466d5700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96400991a0 con 0x7f9640097530 2023-09-30T01:59:08.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f96466d5700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:08.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f96466d5700 10 monclient(hunting): authenticate will time out at 1437.575713s 2023-09-30T01:59:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f9644c72700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f9640097530 0x7f964013a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f963ffff700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9640096a50 0x7f96401327a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42798/0 (socket says 172.21.15.7:42798) 2023-09-30T01:59:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f963ffff700 1 -- 172.21.15.7:0/1173072627 learned_addr learned my addr 172.21.15.7:0/1173072627 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:59:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f9644c72700 10 monclient(hunting): get_auth_request con 0x7f9640097530 auth_method 0 2023-09-30T01:59:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f9644c72700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f9644c72700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f9644c72700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f9644c72700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f9644c72700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.510+0000 7f9644c72700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.511+0000 7f9644c72700 10 monclient(hunting): handle_auth_done global_id 4665 payload 274 2023-09-30T01:59:08.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.511+0000 7f9644c72700 1 -- 172.21.15.7:0/1173072627 >> v1:172.21.15.50:6789/0 conn(0x7f96400a2390 legacy=0x7f9640096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:08.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.511+0000 7f9644c72700 1 -- 172.21.15.7:0/1173072627 >> v1:172.21.15.7:6789/0 conn(0x7f9640096a50 legacy=0x7f96401327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:08.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.511+0000 7f9644c72700 10 monclient: _finish_hunting 0 2023-09-30T01:59:08.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.511+0000 7f9644c72700 1 monclient: found mon.noname-a 2023-09-30T01:59:08.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.511+0000 7f9644c72700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:08.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.511+0000 7f9644c72700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:08.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.511+0000 7f9644c72700 1 -- 172.21.15.7:0/1173072627 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9640099480 con 0x7f9640097530 2023-09-30T01:59:08.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f9644c72700 1 -- 172.21.15.7:0/1173072627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9640097530 msgr2=0x7f964013a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:08.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f9644c72700 1 --2- 172.21.15.7:0/1173072627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9640097530 0x7f964013a730 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9638009e60 tx=0x7f9638009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a258783cb31c6ab4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:08.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f963effd700 1 -- 172.21.15.7:0/1173072627 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963800e070 con 0x7f9640097530 2023-09-30T01:59:08.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f963effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:08.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f963effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:08.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f963effd700 10 monclient: dump: 2023-09-30T01:59:08.521 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:08.521 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:08.521 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:08.521 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:08.522 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:08.522 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:08.522 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:08.522 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:08.522 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:08.522 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:08.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f963effd700 10 monclient: _finish_auth 0 2023-09-30T01:59:08.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f963effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:08.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f963effd700 1 -- 172.21.15.7:0/1173072627 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9638012430 con 0x7f9640097530 2023-09-30T01:59:08.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f96466d5700 5 monclient: authenticate success, global_id 4665 2023-09-30T01:59:08.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f96466d5700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:08.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f963effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:08.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f96466d5700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:08.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f963effd700 1 -- 172.21.15.7:0/1173072627 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963800bea0 con 0x7f9640097530 2023-09-30T01:59:08.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f96466d5700 4 set_mon_vals no callback set 2023-09-30T01:59:08.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f963effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:08.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f963effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:08.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f963effd700 10 monclient: dump: 2023-09-30T01:59:08.523 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:08.523 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:08.523 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:08.523 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:08.523 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:08.523 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:08.523 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:08.524 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:08.524 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:08.524 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:08.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f96466d5700 10 monclient: shutdown 2023-09-30T01:59:08.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f96466d5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:08.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f96466d5700 1 -- 172.21.15.7:0/1173072627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9640097530 msgr2=0x7f964013a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:08.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.513+0000 7f96466d5700 1 --2- 172.21.15.7:0/1173072627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9640097530 0x7f964013a730 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9638009e60 tx=0x7f9638009780 comp rx=0 tx=0).stop 2023-09-30T01:59:08.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 1 -- 172.21.15.7:0/1173072627 shutdown_connections 2023-09-30T01:59:08.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 1 --2- 172.21.15.7:0/1173072627 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9640097530 0x7f964013a730 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:08.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 1 -- 172.21.15.7:0/1173072627 >> 172.21.15.7:0/1173072627 conn(0x7f9640070cf0 msgr2=0x7f96400710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:08.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 1 -- 172.21.15.7:0/1173072627 shutdown_connections 2023-09-30T01:59:08.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 1 -- 172.21.15.7:0/1173072627 wait complete. 2023-09-30T01:59:08.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 10 monclient: build_initial_monmap 2023-09-30T01:59:08.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 10 monclient: monmap: 2023-09-30T01:59:08.525 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:08.525 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:08.525 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:08.525 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:08.525 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:08.525 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:08.525 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:08.525 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:08.525 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:08.526 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:08.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 1 Processor -- start 2023-09-30T01:59:08.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 1 -- start start 2023-09-30T01:59:08.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 10 monclient: init 2023-09-30T01:59:08.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:08.527 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:59:08.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9640096a50 0x7f964009a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:08.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 10 monclient(hunting): picked mon.noname-a con 0x7f9640096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:08.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9640097530 0x7f964009abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:08.528 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 10 monclient(hunting): picked mon.noname-b con 0x7f9640097530 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:08.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f96400a2390 0x7f964009b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:08.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 10 monclient(hunting): picked mon.noname-c con 0x7f96400a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:08.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:08.529 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96400a5310 con 0x7f9640096a50 2023-09-30T01:59:08.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:08.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964009de10 con 0x7f9640097530 2023-09-30T01:59:08.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:08.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964009e0f0 con 0x7f96400a2390 2023-09-30T01:59:08.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:08.530 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f96466d5700 10 monclient(hunting): authenticate will time out at 1437.580628s 2023-09-30T01:59:08.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f963ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9640096a50 0x7f964009a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:08.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f963ffff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9640096a50 0x7f964009a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38574/0 (socket says 172.21.15.7:38574) 2023-09-30T01:59:08.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f963ffff700 1 -- 172.21.15.7:0/2133034908 learned_addr learned my addr 172.21.15.7:0/2133034908 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:08.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f963ffff700 10 monclient(hunting): get_auth_request con 0x7f9640096a50 auth_method 0 2023-09-30T01:59:08.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f963ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:08.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f963ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:08.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f963ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:08.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f9644c72700 1 --2- 172.21.15.7:0/2133034908 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f96400a2390 0x7f964009b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:08.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f9644c72700 10 monclient(hunting): get_auth_request con 0x7f96400a2390 auth_method 0 2023-09-30T01:59:08.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f9644c72700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:08.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f9644c72700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:08.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f9644c72700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:08.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f963ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:08.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f963ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:08.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f963ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:08.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f9644c72700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:08.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f9644c72700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:08.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f9644c72700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:08.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f963f7fe700 1 --2- 172.21.15.7:0/2133034908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9640097530 0x7f964009abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:08.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f963f7fe700 10 monclient(hunting): get_auth_request con 0x7f9640097530 auth_method 0 2023-09-30T01:59:08.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f963f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:08.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f963f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:08.532 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.515+0000 7f963f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:08.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.516+0000 7f9644c72700 10 monclient(hunting): handle_auth_done global_id 4682 payload 824 2023-09-30T01:59:08.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.516+0000 7f9644c72700 1 -- 172.21.15.7:0/2133034908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9640097530 msgr2=0x7f964009abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:08.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.516+0000 7f9644c72700 1 --2- 172.21.15.7:0/2133034908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9640097530 0x7f964009abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:08.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.516+0000 7f9644c72700 1 -- 172.21.15.7:0/2133034908 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9640096a50 msgr2=0x7f964009a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:08.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.516+0000 7f9644c72700 1 --2- 172.21.15.7:0/2133034908 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9640096a50 0x7f964009a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:08.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.516+0000 7f9644c72700 10 monclient: _finish_hunting 0 2023-09-30T01:59:08.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.516+0000 7f9644c72700 1 monclient: found mon.noname-c 2023-09-30T01:59:08.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.516+0000 7f9644c72700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:08.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.516+0000 7f9644c72700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:59:08.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.516+0000 7f9644c72700 1 -- 172.21.15.7:0/2133034908 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9638009a20 con 0x7f96400a2390 2023-09-30T01:59:08.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.516+0000 7f963ffff700 1 --2- 172.21.15.7:0/2133034908 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9640096a50 0x7f964009a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:08.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f9644c72700 1 -- 172.21.15.7:0/2133034908 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f96400a2390 msgr2=0x7f964009b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:08.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f9644c72700 1 --2- 172.21.15.7:0/2133034908 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f96400a2390 0x7f964009b130 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f96400a2b00 tx=0x7f9638018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:08.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f9637fff700 1 -- 172.21.15.7:0/2133034908 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9638026430 con 0x7f96400a2390 2023-09-30T01:59:08.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f9637fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:08.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f9637fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:59:08.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f9637fff700 10 monclient: dump: 2023-09-30T01:59:08.534 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:08.534 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:08.534 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:08.535 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:08.535 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:08.535 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:08.535 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:08.535 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:08.535 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:08.535 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:08.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f9637fff700 10 monclient: _finish_auth 0 2023-09-30T01:59:08.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f9637fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:08.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f96466d5700 5 monclient: authenticate success, global_id 4682 2023-09-30T01:59:08.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f9637fff700 1 -- 172.21.15.7:0/2133034908 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9638021070 con 0x7f96400a2390 2023-09-30T01:59:08.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f96466d5700 10 monclient: _renew_subs 2023-09-30T01:59:08.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f96466d5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:08.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f96466d5700 1 -- 172.21.15.7:0/2133034908 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f964009c2e0 con 0x7f96400a2390 2023-09-30T01:59:08.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f9637fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:08.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f9637fff700 1 -- 172.21.15.7:0/2133034908 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96380267e0 con 0x7f96400a2390 2023-09-30T01:59:08.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f9637fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:08.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f9637fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:08.536 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f9637fff700 10 monclient: dump: 2023-09-30T01:59:08.536 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:08.537 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:08.537 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:08.537 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:08.537 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:08.537 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:08.537 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:08.537 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:08.537 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:08.537 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:08.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f96466d5700 10 monclient: _renew_subs 2023-09-30T01:59:08.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f963e7fc700 4 set_mon_vals no callback set 2023-09-30T01:59:08.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f96466d5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:08.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.518+0000 7f96466d5700 1 -- 172.21.15.7:0/2133034908 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96400a1bd0 con 0x7f96400a2390 2023-09-30T01:59:08.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.519+0000 7f9637fff700 1 -- 172.21.15.7:0/2133034908 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f9638023030 con 0x7f96400a2390 2023-09-30T01:59:08.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.519+0000 7f96466d5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:08.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.519+0000 7f96466d5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:08.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.519+0000 7f96466d5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:08.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.519+0000 7f96466d5700 1 -- 172.21.15.7:0/2133034908 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96400a4770 con 0x7f96400a2390 2023-09-30T01:59:08.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.519+0000 7f9637fff700 1 --2- 172.21.15.7:0/2133034908 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f96200357e0 0x7f9620037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:08.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.519+0000 7f9637fff700 1 -- 172.21.15.7:0/2133034908 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f963804a470 con 0x7f96400a2390 2023-09-30T01:59:08.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.519+0000 7f963ffff700 1 --2- 172.21.15.7:0/2133034908 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f96200357e0 0x7f9620037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:08.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.519+0000 7f963ffff700 10 monclient: get_auth_request con 0x7f96200357e0 auth_method 0 2023-09-30T01:59:08.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.522+0000 7f963ffff700 1 --2- 172.21.15.7:0/2133034908 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f96200357e0 0x7f9620037c90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f96300052e0 tx=0x7f9630004a80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:08.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.524+0000 7f9637fff700 1 -- 172.21.15.7:0/2133034908 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9638025790 con 0x7f96400a2390 2023-09-30T01:59:08.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.524+0000 7f9637fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:08.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.524+0000 7f9637fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:08.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.757+0000 7f96466d5700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:59:08.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.757+0000 7f96466d5700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:08.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.757+0000 7f96466d5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:08.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.757+0000 7f96466d5700 1 -- 172.21.15.7:0/2133034908 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f96400a4770 con 0x7f96400a2390 2023-09-30T01:59:08.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.757+0000 7f9637fff700 1 -- 172.21.15.7:0/2133034908 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f96380253d0 con 0x7f96400a2390 2023-09-30T01:59:08.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.757+0000 7f9637fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:08.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.757+0000 7f9637fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:59:08.758 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:59:08.758 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:08.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.759+0000 7f96466d5700 1 -- 172.21.15.7:0/2133034908 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f96200357e0 msgr2=0x7f9620037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:08.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.759+0000 7f96466d5700 1 --2- 172.21.15.7:0/2133034908 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f96200357e0 0x7f9620037c90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f96300052e0 tx=0x7f9630004a80 comp rx=0 tx=0).stop 2023-09-30T01:59:08.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.759+0000 7f96466d5700 10 monclient: shutdown 2023-09-30T01:59:08.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.759+0000 7f96466d5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:08.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.759+0000 7f96466d5700 1 -- 172.21.15.7:0/2133034908 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f96400a2390 msgr2=0x7f964009b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:08.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.759+0000 7f96466d5700 1 --2- 172.21.15.7:0/2133034908 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f96400a2390 0x7f964009b130 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f96400a2b00 tx=0x7f9638018000 comp rx=0 tx=0).stop 2023-09-30T01:59:08.760 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.759+0000 7f96466d5700 1 -- 172.21.15.7:0/2133034908 shutdown_connections 2023-09-30T01:59:08.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.759+0000 7f96466d5700 1 --2- 172.21.15.7:0/2133034908 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f96200357e0 0x7f9620037c90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:08.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.759+0000 7f96466d5700 1 --2- 172.21.15.7:0/2133034908 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9640096a50 0x7f964009a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:08.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.759+0000 7f96466d5700 1 --2- 172.21.15.7:0/2133034908 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f96400a2390 0x7f964009b130 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:08.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.759+0000 7f96466d5700 1 --2- 172.21.15.7:0/2133034908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9640097530 0x7f964009abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:08.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.759+0000 7f96466d5700 1 -- 172.21.15.7:0/2133034908 >> 172.21.15.7:0/2133034908 conn(0x7f9640070cf0 msgr2=0x7f96401312c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:08.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.759+0000 7f96466d5700 1 -- 172.21.15.7:0/2133034908 shutdown_connections 2023-09-30T01:59:08.761 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:08.759+0000 7f96466d5700 1 -- 172.21.15.7:0/2133034908 wait complete. 2023-09-30T01:59:11.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.901+0000 7f6bbe687700 10 monclient: get_monmap_and_config 2023-09-30T01:59:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.901+0000 7f6bbe687700 10 monclient: build_initial_monmap 2023-09-30T01:59:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.901+0000 7f6bbe687700 10 monclient: monmap: 2023-09-30T01:59:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:11.902845+0000 2023-09-30T01:59:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:11.902845+0000 2023-09-30T01:59:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:11.903 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:11.903 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:11.903 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:11.903 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:11.903 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:11.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.902+0000 7f6bbe687700 1 Processor -- start 2023-09-30T01:59:11.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.902+0000 7f6bbe687700 1 -- start start 2023-09-30T01:59:11.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.902+0000 7f6bbe687700 10 monclient: init 2023-09-30T01:59:11.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbe687700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:11.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbe687700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2023-09-30T01:59:11.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbe687700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6bb8098540 0x7f6bb81329e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:11.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbe687700 10 monclient(hunting): picked mon.noname-c con 0x7f6bb8098540 addr v2:172.21.15.50:3300/0 2023-09-30T01:59:11.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbe687700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bb8098910 0x7f6bb8098ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:11.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbe687700 10 monclient(hunting): picked mon.noname-b con 0x7f6bb8098910 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:11.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbe687700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6bb812ebc0 0x7f6bb812f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:11.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbe687700 10 monclient(hunting): picked mon.noname-a con 0x7f6bb812ebc0 addr v2:172.21.15.7:3300/0 2023-09-30T01:59:11.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbe687700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:11.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbe687700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb80a0f40 con 0x7f6bb812ebc0 2023-09-30T01:59:11.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbe687700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:11.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbe687700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb8069f10 con 0x7f6bb8098540 2023-09-30T01:59:11.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbe687700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:11.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbe687700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb80a05c0 con 0x7f6bb8098910 2023-09-30T01:59:11.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbe687700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:11.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbe687700 10 monclient(hunting): authenticate will time out at 1440.968133s 2023-09-30T01:59:11.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbce84700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bb8098910 0x7f6bb8098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:11.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbde86700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6bb812ebc0 0x7f6bb812f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:11.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbde86700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f6bb812ebc0 0x7f6bb812f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38588/0 (socket says 172.21.15.7:38588) 2023-09-30T01:59:11.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbce84700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bb8098910 0x7f6bb8098ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:45284/0 (socket says 172.21.15.7:45284) 2023-09-30T01:59:11.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbde86700 1 -- 172.21.15.7:0/2431553328 learned_addr learned my addr 172.21.15.7:0/2431553328 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:11.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbde86700 10 monclient(hunting): get_auth_request con 0x7f6bb812ebc0 auth_method 0 2023-09-30T01:59:11.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbde86700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:11.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbde86700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:11.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbde86700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:11.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbce84700 10 monclient(hunting): get_auth_request con 0x7f6bb8098910 auth_method 0 2023-09-30T01:59:11.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbce84700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:11.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbce84700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:11.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbce84700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:11.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbde86700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:11.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbde86700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:11.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbde86700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:11.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbce84700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:11.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbce84700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:11.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbce84700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:11.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbd685700 1 --2- 172.21.15.7:0/2431553328 >> v2:172.21.15.50:3300/0 conn(0x7f6bb8098540 0x7f6bb81329e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:11.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbce84700 10 monclient(hunting): handle_auth_done global_id 4685 payload 274 2023-09-30T01:59:11.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbce84700 1 -- 172.21.15.7:0/2431553328 >> v2:172.21.15.50:3300/0 conn(0x7f6bb8098540 msgr2=0x7f6bb81329e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:11.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbce84700 1 --2- 172.21.15.7:0/2431553328 >> v2:172.21.15.50:3300/0 conn(0x7f6bb8098540 0x7f6bb81329e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:11.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbce84700 1 -- 172.21.15.7:0/2431553328 >> v2:172.21.15.7:3300/0 conn(0x7f6bb812ebc0 msgr2=0x7f6bb812f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:11.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbce84700 1 --2- 172.21.15.7:0/2431553328 >> v2:172.21.15.7:3300/0 conn(0x7f6bb812ebc0 0x7f6bb812f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:11.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbce84700 10 monclient: _finish_hunting 0 2023-09-30T01:59:11.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbce84700 1 monclient: found mon.noname-b 2023-09-30T01:59:11.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbce84700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:11.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbce84700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:11.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbce84700 1 -- 172.21.15.7:0/2431553328 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb8096a50 con 0x7f6bb8098910 2023-09-30T01:59:11.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbd685700 10 monclient: get_auth_request con 0x7f6bb8098540 auth_method 0 2023-09-30T01:59:11.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbde86700 1 --2- 172.21.15.7:0/2431553328 >> v2:172.21.15.7:3300/0 conn(0x7f6bb812ebc0 0x7f6bb812f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:11.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.903+0000 7f6bbd685700 1 --2- 172.21.15.7:0/2431553328 >> v2:172.21.15.50:3300/0 conn(0x7f6bb8098540 0x7f6bb81329e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:59:11.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bbce84700 1 -- 172.21.15.7:0/2431553328 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bb8098910 msgr2=0x7f6bb8098ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:11.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bbce84700 1 --2- 172.21.15.7:0/2431553328 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bb8098910 0x7f6bb8098ce0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8071cb0 tx=0x7f6bac0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ed7a5c9a42e66cf server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:11.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bb7fff700 1 -- 172.21.15.7:0/2431553328 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bac011840 con 0x7f6bb8098910 2023-09-30T01:59:11.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bb7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:11.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bb7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:11.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bb7fff700 10 monclient: dump: 2023-09-30T01:59:11.918 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:11.918 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:11.918 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:11.918 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:11.918 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:11.918 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:11.918 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:11.918 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:11.919 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:11.919 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:11.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bb7fff700 10 monclient: _finish_auth 0 2023-09-30T01:59:11.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bb7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:11.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bbe687700 5 monclient: authenticate success, global_id 4685 2023-09-30T01:59:11.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bb7fff700 1 -- 172.21.15.7:0/2431553328 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bac0119a0 con 0x7f6bb8098910 2023-09-30T01:59:11.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bbe687700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:11.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bb7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:11.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bbe687700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:11.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bbe687700 4 set_mon_vals no callback set 2023-09-30T01:59:11.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bb7fff700 1 -- 172.21.15.7:0/2431553328 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bac011d40 con 0x7f6bb8098910 2023-09-30T01:59:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bb7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bb7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bb7fff700 10 monclient: dump: 2023-09-30T01:59:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:11.920 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:11.921 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:11.921 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:11.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bbe687700 10 monclient: shutdown 2023-09-30T01:59:11.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bbe687700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:11.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bbe687700 1 -- 172.21.15.7:0/2431553328 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bb8098910 msgr2=0x7f6bb8098ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:11.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bbe687700 1 --2- 172.21.15.7:0/2431553328 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bb8098910 0x7f6bb8098ce0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8071cb0 tx=0x7f6bac0091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:11.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bbe687700 1 -- 172.21.15.7:0/2431553328 shutdown_connections 2023-09-30T01:59:11.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bbe687700 1 --2- 172.21.15.7:0/2431553328 >> v2:172.21.15.7:3300/0 conn(0x7f6bb812ebc0 0x7f6bb812f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:11.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bbe687700 1 --2- 172.21.15.7:0/2431553328 >> v2:172.21.15.50:3300/0 conn(0x7f6bb8098540 0x7f6bb81329e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:11.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bbe687700 1 --2- 172.21.15.7:0/2431553328 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bb8098910 0x7f6bb8098ce0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:11.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bbe687700 1 -- 172.21.15.7:0/2431553328 >> 172.21.15.7:0/2431553328 conn(0x7f6bb8071830 msgr2=0x7f6bb80a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:11.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bbe687700 1 -- 172.21.15.7:0/2431553328 shutdown_connections 2023-09-30T01:59:11.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.906+0000 7f6bbe687700 1 -- 172.21.15.7:0/2431553328 wait complete. 2023-09-30T01:59:11.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.907+0000 7f6bbe687700 10 monclient: build_initial_monmap 2023-09-30T01:59:11.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.907+0000 7f6bbe687700 10 monclient: monmap: 2023-09-30T01:59:11.922 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:11.922 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:11.922 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:11.922 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:11.922 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:11.922 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:11.922 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:11.923 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:11.923 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:11.923 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:11.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.907+0000 7f6bbe687700 1 Processor -- start 2023-09-30T01:59:11.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.907+0000 7f6bbe687700 1 -- start start 2023-09-30T01:59:11.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.907+0000 7f6bbe687700 10 monclient: init 2023-09-30T01:59:11.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbe687700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:11.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbe687700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:59:11.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbe687700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bb8098540 0x7f6bb8134ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:11.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbe687700 10 monclient(hunting): picked mon.noname-b con 0x7f6bb8098540 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:11.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbe687700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bb8098910 0x7f6bb812dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:11.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbe687700 10 monclient(hunting): picked mon.noname-a con 0x7f6bb8098910 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:11.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbe687700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bb812ebc0 0x7f6bb80992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:11.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbe687700 10 monclient(hunting): picked mon.noname-c con 0x7f6bb812ebc0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:11.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbe687700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:11.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbe687700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb81327b0 con 0x7f6bb8098910 2023-09-30T01:59:11.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbe687700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:11.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbe687700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb806adb0 con 0x7f6bb8098540 2023-09-30T01:59:11.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbe687700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:11.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbe687700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb809a890 con 0x7f6bb812ebc0 2023-09-30T01:59:11.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbe687700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:11.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbe687700 10 monclient(hunting): authenticate will time out at 1440.973435s 2023-09-30T01:59:11.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbde86700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bb812ebc0 0x7f6bb80992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:11.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbce84700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bb8098910 0x7f6bb812dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:11.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbde86700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bb812ebc0 0x7f6bb80992e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:45292/0 (socket says 172.21.15.7:45292) 2023-09-30T01:59:11.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbde86700 1 -- 172.21.15.7:0/3229311784 learned_addr learned my addr 172.21.15.7:0/3229311784 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:11.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbde86700 10 monclient(hunting): get_auth_request con 0x7f6bb812ebc0 auth_method 0 2023-09-30T01:59:11.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbde86700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:11.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbde86700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:11.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbde86700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:11.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbce84700 10 monclient(hunting): get_auth_request con 0x7f6bb8098910 auth_method 0 2023-09-30T01:59:11.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbce84700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:11.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbce84700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:11.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbce84700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:11.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbde86700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:11.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbde86700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:11.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbde86700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:11.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbce84700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:11.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbce84700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:11.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbce84700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:11.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbd685700 1 --2- 172.21.15.7:0/3229311784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bb8098540 0x7f6bb8134ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:11.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbde86700 10 monclient(hunting): handle_auth_done global_id 4688 payload 824 2023-09-30T01:59:11.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbde86700 1 -- 172.21.15.7:0/3229311784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bb8098540 msgr2=0x7f6bb8134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:11.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbde86700 1 --2- 172.21.15.7:0/3229311784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bb8098540 0x7f6bb8134ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:11.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbde86700 1 -- 172.21.15.7:0/3229311784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bb8098910 msgr2=0x7f6bb812dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:11.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbde86700 1 --2- 172.21.15.7:0/3229311784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bb8098910 0x7f6bb812dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:11.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbde86700 10 monclient: _finish_hunting 0 2023-09-30T01:59:11.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbde86700 1 monclient: found mon.noname-c 2023-09-30T01:59:11.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbde86700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:11.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbde86700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:59:11.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbde86700 1 -- 172.21.15.7:0/3229311784 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb00094f0 con 0x7f6bb812ebc0 2023-09-30T01:59:11.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbd685700 10 monclient: get_auth_request con 0x7f6bb8098540 auth_method 0 2023-09-30T01:59:11.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.908+0000 7f6bbd685700 1 --2- 172.21.15.7:0/3229311784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bb8098540 0x7f6bb8134ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:59:11.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bbde86700 1 -- 172.21.15.7:0/3229311784 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bb812ebc0 msgr2=0x7f6bb80992e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:11.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bbde86700 1 --2- 172.21.15.7:0/3229311784 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bb812ebc0 0x7f6bb80992e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6bb000ec50 tx=0x7f6bb0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:11.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bb57fa700 1 -- 172.21.15.7:0/3229311784 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb000ca60 con 0x7f6bb812ebc0 2023-09-30T01:59:11.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bb57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:11.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bb57fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:59:11.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bb57fa700 10 monclient: dump: 2023-09-30T01:59:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:11.929 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bb57fa700 10 monclient: _finish_auth 0 2023-09-30T01:59:11.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bb57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:11.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bbe687700 5 monclient: authenticate success, global_id 4688 2023-09-30T01:59:11.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bb57fa700 1 -- 172.21.15.7:0/3229311784 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb000cbc0 con 0x7f6bb812ebc0 2023-09-30T01:59:11.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bb57fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:11.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bb57fa700 1 -- 172.21.15.7:0/3229311784 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0013470 con 0x7f6bb812ebc0 2023-09-30T01:59:11.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bbe687700 10 monclient: _renew_subs 2023-09-30T01:59:11.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bbe687700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:11.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bbe687700 1 -- 172.21.15.7:0/3229311784 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bac009420 con 0x7f6bb812ebc0 2023-09-30T01:59:11.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bb57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:11.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bb77fe700 4 set_mon_vals no callback set 2023-09-30T01:59:11.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bb57fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:11.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bb57fa700 10 monclient: dump: 2023-09-30T01:59:11.931 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:11.931 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:11.931 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:11.931 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:11.931 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:11.931 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:11.931 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:11.931 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:11.931 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:11.931 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:11.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bbe687700 10 monclient: _renew_subs 2023-09-30T01:59:11.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bbe687700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:11.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.911+0000 7f6bbe687700 1 -- 172.21.15.7:0/3229311784 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb809a5d0 con 0x7f6bb812ebc0 2023-09-30T01:59:11.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.912+0000 7f6bbe687700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:11.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.912+0000 7f6bb57fa700 1 -- 172.21.15.7:0/3229311784 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f6bb00138a0 con 0x7f6bb812ebc0 2023-09-30T01:59:11.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.912+0000 7f6bbe687700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:11.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.912+0000 7f6bbe687700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:11.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.912+0000 7f6bbe687700 1 -- 172.21.15.7:0/3229311784 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb8064c10 con 0x7f6bb812ebc0 2023-09-30T01:59:11.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.912+0000 7f6bb57fa700 1 --2- 172.21.15.7:0/3229311784 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6ba40357e0 0x7f6ba4037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:11.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.912+0000 7f6bb57fa700 1 -- 172.21.15.7:0/3229311784 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f6bb0047fa0 con 0x7f6bb812ebc0 2023-09-30T01:59:11.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.912+0000 7f6bbd685700 1 --2- 172.21.15.7:0/3229311784 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6ba40357e0 0x7f6ba4037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:11.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.912+0000 7f6bbd685700 10 monclient: get_auth_request con 0x7f6ba40357e0 auth_method 0 2023-09-30T01:59:11.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.915+0000 7f6bbd685700 1 --2- 172.21.15.7:0/3229311784 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6ba40357e0 0x7f6ba4037c90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800d640 tx=0x7f6ba8007670 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:11.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.917+0000 7f6bb57fa700 1 -- 172.21.15.7:0/3229311784 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6bb001a740 con 0x7f6bb812ebc0 2023-09-30T01:59:11.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.917+0000 7f6bb57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:11.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:11.917+0000 7f6bb57fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:12.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:12.154+0000 7f6bbe687700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:59:12.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:12.154+0000 7f6bbe687700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:12.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:12.154+0000 7f6bbe687700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:12.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:12.154+0000 7f6bbe687700 1 -- 172.21.15.7:0/3229311784 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6bb804bb40 con 0x7f6bb812ebc0 2023-09-30T01:59:12.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:12.155+0000 7f6bb57fa700 1 -- 172.21.15.7:0/3229311784 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6bb0020080 con 0x7f6bb812ebc0 2023-09-30T01:59:12.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:12.155+0000 7f6bb57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:12.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:12.155+0000 7f6bb57fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:59:12.155 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:59:12.156 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:12.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:12.157+0000 7f6bbe687700 1 -- 172.21.15.7:0/3229311784 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6ba40357e0 msgr2=0x7f6ba4037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:12.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:12.157+0000 7f6bbe687700 1 --2- 172.21.15.7:0/3229311784 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6ba40357e0 0x7f6ba4037c90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800d640 tx=0x7f6ba8007670 comp rx=0 tx=0).stop 2023-09-30T01:59:12.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:12.157+0000 7f6bbe687700 10 monclient: shutdown 2023-09-30T01:59:12.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:12.157+0000 7f6bbe687700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:12.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:12.157+0000 7f6bbe687700 1 -- 172.21.15.7:0/3229311784 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bb812ebc0 msgr2=0x7f6bb80992e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:12.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:12.157+0000 7f6bbe687700 1 --2- 172.21.15.7:0/3229311784 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bb812ebc0 0x7f6bb80992e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6bb000ec50 tx=0x7f6bb0009250 comp rx=0 tx=0).stop 2023-09-30T01:59:12.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:12.157+0000 7f6bbe687700 1 -- 172.21.15.7:0/3229311784 shutdown_connections 2023-09-30T01:59:12.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:12.157+0000 7f6bbe687700 1 --2- 172.21.15.7:0/3229311784 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6ba40357e0 0x7f6ba4037c90 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:12.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:12.157+0000 7f6bbe687700 1 --2- 172.21.15.7:0/3229311784 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6bb812ebc0 0x7f6bb80992e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:12.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:12.157+0000 7f6bbe687700 1 --2- 172.21.15.7:0/3229311784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bb8098540 0x7f6bb8134ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:12.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:12.157+0000 7f6bbe687700 1 --2- 172.21.15.7:0/3229311784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6bb8098910 0x7f6bb812dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:12.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:12.157+0000 7f6bbe687700 1 -- 172.21.15.7:0/3229311784 >> 172.21.15.7:0/3229311784 conn(0x7f6bb8071830 msgr2=0x7f6bb80a3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:12.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:12.157+0000 7f6bbe687700 1 -- 172.21.15.7:0/3229311784 shutdown_connections 2023-09-30T01:59:12.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:12.157+0000 7f6bbe687700 1 -- 172.21.15.7:0/3229311784 wait complete. 2023-09-30T01:59:15.170 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:15.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.299+0000 7f23f9a1a700 10 monclient: get_monmap_and_config 2023-09-30T01:59:15.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.299+0000 7f23f9a1a700 10 monclient: build_initial_monmap 2023-09-30T01:59:15.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.299+0000 7f23f9a1a700 10 monclient: monmap: 2023-09-30T01:59:15.299 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:15.299 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:15.299 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:15.300184+0000 2023-09-30T01:59:15.299 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:15.300184+0000 2023-09-30T01:59:15.299 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:15.300 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:15.300 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:15.300 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:15.300 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:15.300 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:15.300 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:15.300 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:15.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.299+0000 7f23f9a1a700 1 Processor -- start 2023-09-30T01:59:15.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.299+0000 7f23f9a1a700 1 -- start start 2023-09-30T01:59:15.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.299+0000 7f23f9a1a700 10 monclient: init 2023-09-30T01:59:15.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f9a1a700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:15.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f9a1a700 10 monclient: _add_conns ranks=[0,3,2,4,1] 2023-09-30T01:59:15.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f9a1a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f23f4096a50 addr v1:172.21.15.7:6789/0 2023-09-30T01:59:15.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f9a1a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f23f4096e20 0x7f23f4097220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:15.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f9a1a700 10 monclient(hunting): picked mon.noname-a con 0x7f23f4096e20 addr v2:172.21.15.7:3300/0 2023-09-30T01:59:15.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f9a1a700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23f4137710 0x7f23f4139af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:15.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f9a1a700 10 monclient(hunting): picked mon.noname-b con 0x7f23f4137710 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:15.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f9a1a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23f4099470 con 0x7f23f4096a50 2023-09-30T01:59:15.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f9a1a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:15.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f9a1a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23f4132d90 con 0x7f23f4096e20 2023-09-30T01:59:15.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f9a1a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:15.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f9a1a700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f4097b40 con 0x7f23f4137710 2023-09-30T01:59:15.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f9a1a700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:15.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f9a1a700 10 monclient(hunting): authenticate will time out at 1444.365274s 2023-09-30T01:59:15.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f2ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f23f4096a50 0x7f23f4132680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33308/0 (socket says 172.21.15.7:33308) 2023-09-30T01:59:15.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f27fc700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f23f4096e20 0x7f23f4097220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:15.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f2ffd700 1 -- 172.21.15.7:0/2222057534 learned_addr learned my addr 172.21.15.7:0/2222057534 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:59:15.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f37fe700 1 --2- 172.21.15.7:0/2222057534 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23f4137710 0x7f23f4139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f27fc700 10 monclient(hunting): get_auth_request con 0x7f23f4096e20 auth_method 0 2023-09-30T01:59:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f27fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f27fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f37fe700 10 monclient(hunting): get_auth_request con 0x7f23f4137710 auth_method 0 2023-09-30T01:59:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:15.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:15.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:15.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:15.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f27fc700 10 monclient(hunting): handle_auth_done global_id 4677 payload 274 2023-09-30T01:59:15.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f27fc700 1 -- 172.21.15.7:0/2222057534 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23f4137710 msgr2=0x7f23f4139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:15.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f27fc700 1 --2- 172.21.15.7:0/2222057534 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23f4137710 0x7f23f4139af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:15.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f27fc700 1 -- 172.21.15.7:0/2222057534 >> v1:172.21.15.7:6789/0 conn(0x7f23f4096a50 legacy=0x7f23f4132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:15.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f27fc700 10 monclient: _finish_hunting 0 2023-09-30T01:59:15.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f27fc700 1 monclient: found mon.noname-a 2023-09-30T01:59:15.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:15.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f27fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:15.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f27fc700 1 -- 172.21.15.7:0/2222057534 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f4099160 con 0x7f23f4096e20 2023-09-30T01:59:15.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.300+0000 7f23f37fe700 1 --2- 172.21.15.7:0/2222057534 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23f4137710 0x7f23f4139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:15.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f27fc700 1 -- 172.21.15.7:0/2222057534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23f4096e20 msgr2=0x7f23f4097220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:15.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f27fc700 1 --2- 172.21.15.7:0/2222057534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23f4096e20 0x7f23f4097220 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f23e80098e0 tx=0x7f23e80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63065e87623815d7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:15.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f1ffb700 1 -- 172.21.15.7:0/2222057534 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e8011840 con 0x7f23f4096e20 2023-09-30T01:59:15.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:15.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:15.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f1ffb700 10 monclient: dump: 2023-09-30T01:59:15.308 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:15.308 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:15.308 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:15.308 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:15.308 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:15.308 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:15.308 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:15.309 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:15.309 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:15.309 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:15.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f1ffb700 10 monclient: _finish_auth 0 2023-09-30T01:59:15.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:15.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f9a1a700 5 monclient: authenticate success, global_id 4677 2023-09-30T01:59:15.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f1ffb700 1 -- 172.21.15.7:0/2222057534 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e80119a0 con 0x7f23f4096e20 2023-09-30T01:59:15.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f9a1a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:15.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:15.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f9a1a700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:15.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f9a1a700 4 set_mon_vals no callback set 2023-09-30T01:59:15.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f1ffb700 1 -- 172.21.15.7:0/2222057534 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e8011d40 con 0x7f23f4096e20 2023-09-30T01:59:15.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:15.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:15.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f1ffb700 10 monclient: dump: 2023-09-30T01:59:15.310 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:15.310 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:15.310 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:15.310 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:15.310 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:15.310 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:15.310 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:15.310 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:15.310 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:15.310 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:15.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f9a1a700 10 monclient: shutdown 2023-09-30T01:59:15.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f9a1a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:15.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f9a1a700 1 -- 172.21.15.7:0/2222057534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23f4096e20 msgr2=0x7f23f4097220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:15.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f9a1a700 1 --2- 172.21.15.7:0/2222057534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23f4096e20 0x7f23f4097220 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f23e80098e0 tx=0x7f23e80091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:15.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f9a1a700 1 -- 172.21.15.7:0/2222057534 shutdown_connections 2023-09-30T01:59:15.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f9a1a700 1 --2- 172.21.15.7:0/2222057534 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23f4137710 0x7f23f4139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:15.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f9a1a700 1 --2- 172.21.15.7:0/2222057534 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23f4096e20 0x7f23f4097220 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:15.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f9a1a700 1 -- 172.21.15.7:0/2222057534 >> 172.21.15.7:0/2222057534 conn(0x7f23f4070cf0 msgr2=0x7f23f40710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:15.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f9a1a700 1 -- 172.21.15.7:0/2222057534 shutdown_connections 2023-09-30T01:59:15.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.303+0000 7f23f9a1a700 1 -- 172.21.15.7:0/2222057534 wait complete. 2023-09-30T01:59:15.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.304+0000 7f23f9a1a700 10 monclient: build_initial_monmap 2023-09-30T01:59:15.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.304+0000 7f23f9a1a700 10 monclient: monmap: 2023-09-30T01:59:15.312 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:15.312 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:15.312 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:15.312 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:15.312 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:15.312 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:15.312 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:15.312 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:15.312 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:15.312 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:15.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.304+0000 7f23f9a1a700 1 Processor -- start 2023-09-30T01:59:15.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.304+0000 7f23f9a1a700 1 -- start start 2023-09-30T01:59:15.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.304+0000 7f23f9a1a700 10 monclient: init 2023-09-30T01:59:15.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f9a1a700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:15.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f9a1a700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:59:15.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f9a1a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23f4096a50 0x7f23f4141f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:15.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f9a1a700 10 monclient(hunting): picked mon.noname-a con 0x7f23f4096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:15.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f9a1a700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23f4096e20 0x7f23f40a1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:15.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f9a1a700 10 monclient(hunting): picked mon.noname-c con 0x7f23f4096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:15.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f9a1a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23f4137710 0x7f23f40a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:15.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f9a1a700 10 monclient(hunting): picked mon.noname-b con 0x7f23f4137710 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:15.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f9a1a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:15.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f9a1a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f409dd10 con 0x7f23f4096a50 2023-09-30T01:59:15.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f9a1a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:15.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f9a1a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f4131e20 con 0x7f23f4137710 2023-09-30T01:59:15.317 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f9a1a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:15.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f9a1a700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f409da30 con 0x7f23f4096e20 2023-09-30T01:59:15.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f9a1a700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:15.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f9a1a700 10 monclient(hunting): authenticate will time out at 1444.370293s 2023-09-30T01:59:15.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f2ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23f4096a50 0x7f23f4141f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:15.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f27fc700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23f4096e20 0x7f23f40a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:15.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f27fc700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23f4096e20 0x7f23f40a1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:44980/0 (socket says 172.21.15.7:44980) 2023-09-30T01:59:15.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f2ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23f4096a50 0x7f23f4141f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36814/0 (socket says 172.21.15.7:36814) 2023-09-30T01:59:15.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f27fc700 1 -- 172.21.15.7:0/3489089157 learned_addr learned my addr 172.21.15.7:0/3489089157 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:15.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f27fc700 10 monclient(hunting): get_auth_request con 0x7f23f4096e20 auth_method 0 2023-09-30T01:59:15.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:15.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f27fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:15.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f27fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:15.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f2ffd700 10 monclient(hunting): get_auth_request con 0x7f23f4096a50 auth_method 0 2023-09-30T01:59:15.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:15.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:15.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:15.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:15.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:15.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:15.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:15.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:15.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:15.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f2ffd700 10 monclient(hunting): handle_auth_done global_id 4680 payload 824 2023-09-30T01:59:15.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f2ffd700 1 -- 172.21.15.7:0/3489089157 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23f4096e20 msgr2=0x7f23f40a1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:15.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f2ffd700 1 --2- 172.21.15.7:0/3489089157 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23f4096e20 0x7f23f40a1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:15.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f2ffd700 1 -- 172.21.15.7:0/3489089157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23f4137710 msgr2=0x7f23f40a1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:15.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f37fe700 1 --2- 172.21.15.7:0/3489089157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23f4137710 0x7f23f40a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:15.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f2ffd700 1 --2- 172.21.15.7:0/3489089157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23f4137710 0x7f23f40a1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:15.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f2ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:59:15.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f2ffd700 1 monclient: found mon.noname-a 2023-09-30T01:59:15.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:15.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f2ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:15.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f2ffd700 1 -- 172.21.15.7:0/3489089157 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ec009a20 con 0x7f23f4096a50 2023-09-30T01:59:15.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.305+0000 7f23f27fc700 1 --2- 172.21.15.7:0/3489089157 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23f4096e20 0x7f23f40a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:15.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.307+0000 7f23f2ffd700 1 -- 172.21.15.7:0/3489089157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23f4096a50 msgr2=0x7f23f4141f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:15.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.307+0000 7f23f2ffd700 1 --2- 172.21.15.7:0/3489089157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23f4096a50 0x7f23f4141f90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f23e400c530 tx=0x7f23e4007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:15.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.308+0000 7f23daffd700 1 -- 172.21.15.7:0/3489089157 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e4005d40 con 0x7f23f4096a50 2023-09-30T01:59:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.308+0000 7f23daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.308+0000 7f23daffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.308+0000 7f23daffd700 10 monclient: dump: 2023-09-30T01:59:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:15.321 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:15.322 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:15.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.308+0000 7f23daffd700 10 monclient: _finish_auth 0 2023-09-30T01:59:15.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.308+0000 7f23daffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:15.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.308+0000 7f23daffd700 1 -- 172.21.15.7:0/3489089157 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e40070d0 con 0x7f23f4096a50 2023-09-30T01:59:15.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.308+0000 7f23f9a1a700 5 monclient: authenticate success, global_id 4680 2023-09-30T01:59:15.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.308+0000 7f23f9a1a700 10 monclient: _renew_subs 2023-09-30T01:59:15.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.308+0000 7f23f9a1a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:15.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.308+0000 7f23f9a1a700 1 -- 172.21.15.7:0/3489089157 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e8009420 con 0x7f23f4096a50 2023-09-30T01:59:15.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.308+0000 7f23daffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:15.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.308+0000 7f23daffd700 1 -- 172.21.15.7:0/3489089157 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e4007490 con 0x7f23f4096a50 2023-09-30T01:59:15.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.308+0000 7f23daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:15.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.308+0000 7f23daffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:15.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.308+0000 7f23daffd700 10 monclient: dump: 2023-09-30T01:59:15.323 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:15.323 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:15.323 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:15.323 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:15.323 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:15.323 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:15.323 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:15.323 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:15.323 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:15.323 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:15.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.308+0000 7f23f17fa700 4 set_mon_vals no callback set 2023-09-30T01:59:15.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.308+0000 7f23f9a1a700 10 monclient: _renew_subs 2023-09-30T01:59:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.308+0000 7f23f9a1a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.308+0000 7f23f9a1a700 1 -- 172.21.15.7:0/3489089157 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f409bc40 con 0x7f23f4096a50 2023-09-30T01:59:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.309+0000 7f23daffd700 1 -- 172.21.15.7:0/3489089157 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f23e401b3b0 con 0x7f23f4096a50 2023-09-30T01:59:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.309+0000 7f23f9a1a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.309+0000 7f23f9a1a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.309+0000 7f23f9a1a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.309+0000 7f23f9a1a700 1 -- 172.21.15.7:0/3489089157 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f40a3040 con 0x7f23f4096a50 2023-09-30T01:59:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.309+0000 7f23daffd700 1 --2- 172.21.15.7:0/3489089157 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f23e00357e0 0x7f23e0037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.309+0000 7f23daffd700 1 -- 172.21.15.7:0/3489089157 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f23e40471f0 con 0x7f23f4096a50 2023-09-30T01:59:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.309+0000 7f23f27fc700 1 --2- 172.21.15.7:0/3489089157 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f23e00357e0 0x7f23e0037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.309+0000 7f23f27fc700 10 monclient: get_auth_request con 0x7f23e00357e0 auth_method 0 2023-09-30T01:59:15.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.312+0000 7f23f27fc700 1 --2- 172.21.15.7:0/3489089157 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f23e00357e0 0x7f23e0037c90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f23f409b8d0 tx=0x7f23e8016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:15.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.313+0000 7f23daffd700 1 -- 172.21.15.7:0/3489089157 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f23e4018a50 con 0x7f23f4096a50 2023-09-30T01:59:15.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.313+0000 7f23daffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:15.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.313+0000 7f23daffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:15.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.554+0000 7f23f9a1a700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:59:15.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.554+0000 7f23f9a1a700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:15.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.554+0000 7f23f9a1a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:15.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.554+0000 7f23f9a1a700 1 -- 172.21.15.7:0/3489089157 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f23f40711e0 con 0x7f23f4096a50 2023-09-30T01:59:15.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.555+0000 7f23daffd700 1 -- 172.21.15.7:0/3489089157 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f23e4017100 con 0x7f23f4096a50 2023-09-30T01:59:15.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.555+0000 7f23daffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:15.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.555+0000 7f23daffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:59:15.556 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:59:15.556 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:15.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.557+0000 7f23f9a1a700 1 -- 172.21.15.7:0/3489089157 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f23e00357e0 msgr2=0x7f23e0037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:15.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.557+0000 7f23f9a1a700 1 --2- 172.21.15.7:0/3489089157 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f23e00357e0 0x7f23e0037c90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f23f409b8d0 tx=0x7f23e8016000 comp rx=0 tx=0).stop 2023-09-30T01:59:15.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.557+0000 7f23f9a1a700 10 monclient: shutdown 2023-09-30T01:59:15.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.557+0000 7f23f9a1a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:15.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.557+0000 7f23f9a1a700 1 -- 172.21.15.7:0/3489089157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23f4096a50 msgr2=0x7f23f4141f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:15.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.557+0000 7f23f9a1a700 1 --2- 172.21.15.7:0/3489089157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23f4096a50 0x7f23f4141f90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f23e400c530 tx=0x7f23e4007f90 comp rx=0 tx=0).stop 2023-09-30T01:59:15.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.557+0000 7f23f9a1a700 1 -- 172.21.15.7:0/3489089157 shutdown_connections 2023-09-30T01:59:15.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.557+0000 7f23f9a1a700 1 --2- 172.21.15.7:0/3489089157 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f23e00357e0 0x7f23e0037c90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:15.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.557+0000 7f23f9a1a700 1 --2- 172.21.15.7:0/3489089157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f23f4096a50 0x7f23f4141f90 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:15.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.557+0000 7f23f9a1a700 1 --2- 172.21.15.7:0/3489089157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23f4137710 0x7f23f40a1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:15.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.557+0000 7f23f9a1a700 1 --2- 172.21.15.7:0/3489089157 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f23f4096e20 0x7f23f40a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:15.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.557+0000 7f23f9a1a700 1 -- 172.21.15.7:0/3489089157 >> 172.21.15.7:0/3489089157 conn(0x7f23f4070cf0 msgr2=0x7f23f412f570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:15.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.557+0000 7f23f9a1a700 1 -- 172.21.15.7:0/3489089157 shutdown_connections 2023-09-30T01:59:15.558 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:15.557+0000 7f23f9a1a700 1 -- 172.21.15.7:0/3489089157 wait complete. 2023-09-30T01:59:18.569 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:18.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.700+0000 7fbabd660700 10 monclient: get_monmap_and_config 2023-09-30T01:59:18.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.700+0000 7fbabd660700 10 monclient: build_initial_monmap 2023-09-30T01:59:18.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.700+0000 7fbabd660700 10 monclient: monmap: 2023-09-30T01:59:18.701 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:18.701 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:18.701 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:18.701698+0000 2023-09-30T01:59:18.701 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:18.701698+0000 2023-09-30T01:59:18.701 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:18.701 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:18.701 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:18.701 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:18.701 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:18.701 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:18.701 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:18.701 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:18.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.701+0000 7fbabd660700 1 Processor -- start 2023-09-30T01:59:18.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.701+0000 7fbabd660700 1 -- start start 2023-09-30T01:59:18.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.701+0000 7fbabd660700 10 monclient: init 2023-09-30T01:59:18.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.701+0000 7fbabd660700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:18.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.701+0000 7fbabd660700 10 monclient: _add_conns ranks=[3,1,2,4,0] 2023-09-30T01:59:18.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.701+0000 7fbabd660700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbab809ba60 0x7fbab8099a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:18.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.701+0000 7fbabd660700 10 monclient(hunting): picked mon.noname-a con 0x7fbab809ba60 addr v2:172.21.15.7:3300/0 2023-09-30T01:59:18.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.701+0000 7fbabd660700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbab8099fd0 addr v1:172.21.15.50:6789/0 2023-09-30T01:59:18.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.701+0000 7fbabd660700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbab809c540 0x7fbab8135ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:18.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.701+0000 7fbabd660700 10 monclient(hunting): picked mon.noname-b con 0x7fbab809c540 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:18.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.701+0000 7fbabd660700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab8099390 con 0x7fbab8099fd0 2023-09-30T01:59:18.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.701+0000 7fbabd660700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:18.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.701+0000 7fbabd660700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbab809d9e0 con 0x7fbab809ba60 2023-09-30T01:59:18.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.701+0000 7fbabd660700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:18.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.701+0000 7fbabd660700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab80990b0 con 0x7fbab809c540 2023-09-30T01:59:18.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.701+0000 7fbabd660700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:18.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.701+0000 7fbabd660700 10 monclient(hunting): authenticate will time out at 1447.766960s 2023-09-30T01:59:18.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbab7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbab809ba60 0x7fbab8099a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:18.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbabce5f700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbab809c540 0x7fbab8135ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:18.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbab7fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fbab809ba60 0x7fbab8099a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36824/0 (socket says 172.21.15.7:36824) 2023-09-30T01:59:18.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbab7fff700 1 -- 172.21.15.7:0/3086917323 learned_addr learned my addr 172.21.15.7:0/3086917323 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:18.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbab7fff700 10 monclient(hunting): get_auth_request con 0x7fbab809ba60 auth_method 0 2023-09-30T01:59:18.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbab7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:18.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbab7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:18.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbab7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:18.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbabce5f700 10 monclient(hunting): get_auth_request con 0x7fbab809c540 auth_method 0 2023-09-30T01:59:18.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbabce5f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:18.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbabce5f700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:18.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbabce5f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:18.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbab7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:18.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbab7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:18.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbab7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:18.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbabce5f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:18.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbabce5f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:18.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbabce5f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:18.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbabce5f700 10 monclient(hunting): handle_auth_done global_id 4697 payload 274 2023-09-30T01:59:18.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbabce5f700 1 -- 172.21.15.7:0/3086917323 >> v2:172.21.15.7:3300/0 conn(0x7fbab809ba60 msgr2=0x7fbab8099a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:18.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbabce5f700 1 --2- 172.21.15.7:0/3086917323 >> v2:172.21.15.7:3300/0 conn(0x7fbab809ba60 0x7fbab8099a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:18.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbabce5f700 1 -- 172.21.15.7:0/3086917323 >> v1:172.21.15.50:6789/0 conn(0x7fbab8099fd0 legacy=0x7fbab809be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:18.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbabce5f700 10 monclient: _finish_hunting 0 2023-09-30T01:59:18.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbabce5f700 1 monclient: found mon.noname-b 2023-09-30T01:59:18.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbabce5f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:18.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbabce5f700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:18.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbabce5f700 1 -- 172.21.15.7:0/3086917323 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab809cd20 con 0x7fbab809c540 2023-09-30T01:59:18.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.702+0000 7fbab7fff700 1 --2- 172.21.15.7:0/3086917323 >> v2:172.21.15.7:3300/0 conn(0x7fbab809ba60 0x7fbab8099a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:18.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.704+0000 7fbabce5f700 1 -- 172.21.15.7:0/3086917323 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbab809c540 msgr2=0x7fbab8135ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:18.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.704+0000 7fbabce5f700 1 --2- 172.21.15.7:0/3086917323 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbab809c540 0x7fbab8135ab0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbab0009e60 tx=0x7fbab0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e1c0f58cff8c454 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:18.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.704+0000 7fbab6ffd700 1 -- 172.21.15.7:0/3086917323 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab000e070 con 0x7fbab809c540 2023-09-30T01:59:18.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.704+0000 7fbab6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:18.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.704+0000 7fbab6ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:18.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.704+0000 7fbab6ffd700 10 monclient: dump: 2023-09-30T01:59:18.710 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:18.710 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:18.710 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:18.710 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:18.710 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:18.710 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:18.711 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:18.711 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:18.711 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:18.711 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:18.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.704+0000 7fbab6ffd700 10 monclient: _finish_auth 0 2023-09-30T01:59:18.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.704+0000 7fbab6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:18.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.704+0000 7fbab6ffd700 1 -- 172.21.15.7:0/3086917323 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab000bd30 con 0x7fbab809c540 2023-09-30T01:59:18.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.704+0000 7fbabd660700 5 monclient: authenticate success, global_id 4697 2023-09-30T01:59:18.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.704+0000 7fbabd660700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:18.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.704+0000 7fbab6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:18.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.704+0000 7fbabd660700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:18.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.704+0000 7fbab6ffd700 1 -- 172.21.15.7:0/3086917323 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab000bea0 con 0x7fbab809c540 2023-09-30T01:59:18.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.704+0000 7fbab6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:18.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.704+0000 7fbabd660700 4 set_mon_vals no callback set 2023-09-30T01:59:18.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.704+0000 7fbab6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:18.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.704+0000 7fbab6ffd700 10 monclient: dump: 2023-09-30T01:59:18.712 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:18.712 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:18.712 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:18.712 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:18.712 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:18.712 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:18.712 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:18.712 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:18.713 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:18.713 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:18.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.705+0000 7fbabd660700 10 monclient: shutdown 2023-09-30T01:59:18.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.705+0000 7fbabd660700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:18.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.705+0000 7fbabd660700 1 -- 172.21.15.7:0/3086917323 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbab809c540 msgr2=0x7fbab8135ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:18.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.705+0000 7fbabd660700 1 --2- 172.21.15.7:0/3086917323 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbab809c540 0x7fbab8135ab0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbab0009e60 tx=0x7fbab0009780 comp rx=0 tx=0).stop 2023-09-30T01:59:18.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.705+0000 7fbabd660700 1 -- 172.21.15.7:0/3086917323 shutdown_connections 2023-09-30T01:59:18.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.705+0000 7fbabd660700 1 --2- 172.21.15.7:0/3086917323 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbab809c540 0x7fbab8135ab0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:18.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.705+0000 7fbabd660700 1 --2- 172.21.15.7:0/3086917323 >> v2:172.21.15.7:3300/0 conn(0x7fbab809ba60 0x7fbab8099a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:18.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.705+0000 7fbabd660700 1 -- 172.21.15.7:0/3086917323 >> 172.21.15.7:0/3086917323 conn(0x7fbab80a1230 msgr2=0x7fbab80a1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:18.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.705+0000 7fbabd660700 1 -- 172.21.15.7:0/3086917323 shutdown_connections 2023-09-30T01:59:18.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.705+0000 7fbabd660700 1 -- 172.21.15.7:0/3086917323 wait complete. 2023-09-30T01:59:18.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.705+0000 7fbabd660700 10 monclient: build_initial_monmap 2023-09-30T01:59:18.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.706+0000 7fbabd660700 10 monclient: monmap: 2023-09-30T01:59:18.714 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:18.714 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:18.714 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:18.714 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:18.714 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:18.715 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:18.715 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:18.715 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:18.715 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:18.715 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:18.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.706+0000 7fbabd660700 1 Processor -- start 2023-09-30T01:59:18.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.706+0000 7fbabd660700 1 -- start start 2023-09-30T01:59:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.706+0000 7fbabd660700 10 monclient: init 2023-09-30T01:59:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbabd660700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbabd660700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:59:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbabd660700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbab8099fd0 0x7fbab8136410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbabd660700 10 monclient(hunting): picked mon.noname-a con 0x7fbab8099fd0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbabd660700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbab809ba60 0x7fbab81415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbabd660700 10 monclient(hunting): picked mon.noname-c con 0x7fbab809ba60 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbabd660700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbab809c540 0x7fbab8141b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbabd660700 10 monclient(hunting): picked mon.noname-b con 0x7fbab809c540 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbabd660700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:18.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbabd660700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab809f2c0 con 0x7fbab8099fd0 2023-09-30T01:59:18.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbabd660700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:18.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbabd660700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab8136950 con 0x7fbab809c540 2023-09-30T01:59:18.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbabd660700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:18.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbabd660700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab8135400 con 0x7fbab809ba60 2023-09-30T01:59:18.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbabd660700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:18.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbabd660700 10 monclient(hunting): authenticate will time out at 1447.772122s 2023-09-30T01:59:18.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbab8099fd0 0x7fbab8136410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:18.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab7fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbab8099fd0 0x7fbab8136410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36830/0 (socket says 172.21.15.7:36830) 2023-09-30T01:59:18.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab77fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbab809ba60 0x7fbab81415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:18.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab7fff700 1 -- 172.21.15.7:0/312689086 learned_addr learned my addr 172.21.15.7:0/312689086 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:18.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab7fff700 10 monclient(hunting): get_auth_request con 0x7fbab8099fd0 auth_method 0 2023-09-30T01:59:18.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:18.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:18.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:18.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab77fe700 10 monclient(hunting): get_auth_request con 0x7fbab809ba60 auth_method 0 2023-09-30T01:59:18.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:18.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:18.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:18.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:18.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:18.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:18.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:18.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:18.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:18.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbabce5f700 1 --2- 172.21.15.7:0/312689086 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbab809c540 0x7fbab8141b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:18.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab7fff700 10 monclient(hunting): handle_auth_done global_id 4686 payload 824 2023-09-30T01:59:18.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab7fff700 1 -- 172.21.15.7:0/312689086 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbab809ba60 msgr2=0x7fbab81415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:18.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab7fff700 1 --2- 172.21.15.7:0/312689086 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbab809ba60 0x7fbab81415c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:18.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab7fff700 1 -- 172.21.15.7:0/312689086 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbab809c540 msgr2=0x7fbab8141b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:18.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab7fff700 1 --2- 172.21.15.7:0/312689086 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbab809c540 0x7fbab8141b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:18.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab7fff700 10 monclient: _finish_hunting 0 2023-09-30T01:59:18.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab7fff700 1 monclient: found mon.noname-a 2023-09-30T01:59:18.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:18.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab7fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:18.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab7fff700 1 -- 172.21.15.7:0/312689086 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa00094f0 con 0x7fbab8099fd0 2023-09-30T01:59:18.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbabce5f700 10 monclient: get_auth_request con 0x7fbab809c540 auth_method 0 2023-09-30T01:59:18.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbabce5f700 1 --2- 172.21.15.7:0/312689086 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbab809c540 0x7fbab8141b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:59:18.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.707+0000 7fbab77fe700 1 --2- 172.21.15.7:0/312689086 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbab809ba60 0x7fbab81415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:18.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.709+0000 7fbab7fff700 1 -- 172.21.15.7:0/312689086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbab8099fd0 msgr2=0x7fbab8136410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:18.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.709+0000 7fbab7fff700 1 --2- 172.21.15.7:0/312689086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbab8099fd0 0x7fbab8136410 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbaa000ec50 tx=0x7fbaa0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:18.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.709+0000 7fbaaffff700 1 -- 172.21.15.7:0/312689086 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa00097d0 con 0x7fbab8099fd0 2023-09-30T01:59:18.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.709+0000 7fbaaffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:18.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.709+0000 7fbaaffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:18.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.709+0000 7fbaaffff700 10 monclient: dump: 2023-09-30T01:59:18.724 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:18.724 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:18.724 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:18.725 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:18.725 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:18.725 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:18.725 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:18.725 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:18.725 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:18.725 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:18.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.709+0000 7fbaaffff700 10 monclient: _finish_auth 0 2023-09-30T01:59:18.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.709+0000 7fbaaffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:18.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.709+0000 7fbaaffff700 1 -- 172.21.15.7:0/312689086 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa000cb30 con 0x7fbab8099fd0 2023-09-30T01:59:18.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.709+0000 7fbabd660700 5 monclient: authenticate success, global_id 4686 2023-09-30T01:59:18.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.709+0000 7fbabd660700 10 monclient: _renew_subs 2023-09-30T01:59:18.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.709+0000 7fbabd660700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:18.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.709+0000 7fbabd660700 1 -- 172.21.15.7:0/312689086 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab0009a20 con 0x7fbab8099fd0 2023-09-30T01:59:18.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.709+0000 7fbaaffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:18.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.709+0000 7fbaaffff700 1 -- 172.21.15.7:0/312689086 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa0013470 con 0x7fbab8099fd0 2023-09-30T01:59:18.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.710+0000 7fbaaffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:18.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.710+0000 7fbaaffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:18.726 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.710+0000 7fbaaffff700 10 monclient: dump: 2023-09-30T01:59:18.726 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:18.726 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:18.726 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:18.726 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:18.726 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:18.727 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:18.727 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:18.727 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:18.727 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:18.727 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:18.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.710+0000 7fbab67fc700 4 set_mon_vals no callback set 2023-09-30T01:59:18.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.710+0000 7fbabd660700 10 monclient: _renew_subs 2023-09-30T01:59:18.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.710+0000 7fbabd660700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:18.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.710+0000 7fbabd660700 1 -- 172.21.15.7:0/312689086 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab8134510 con 0x7fbab8099fd0 2023-09-30T01:59:18.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.710+0000 7fbaaffff700 1 -- 172.21.15.7:0/312689086 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fbaa0013820 con 0x7fbab8099fd0 2023-09-30T01:59:18.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.710+0000 7fbabd660700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:18.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.710+0000 7fbabd660700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:18.727 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.710+0000 7fbabd660700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:18.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.710+0000 7fbabd660700 1 -- 172.21.15.7:0/312689086 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbab8135b80 con 0x7fbab8099fd0 2023-09-30T01:59:18.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.710+0000 7fbaaffff700 1 --2- 172.21.15.7:0/312689086 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fba980357e0 0x7fba98037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:18.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.711+0000 7fbaaffff700 1 -- 172.21.15.7:0/312689086 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fbaa0048410 con 0x7fbab8099fd0 2023-09-30T01:59:18.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.711+0000 7fbab77fe700 1 --2- 172.21.15.7:0/312689086 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fba980357e0 0x7fba98037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:18.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.711+0000 7fbab77fe700 10 monclient: get_auth_request con 0x7fba980357e0 auth_method 0 2023-09-30T01:59:18.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.713+0000 7fbab77fe700 1 --2- 172.21.15.7:0/312689086 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fba980357e0 0x7fba98037c90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbab8134cc0 tx=0x7fbaa8007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:18.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.715+0000 7fbaaffff700 1 -- 172.21.15.7:0/312689086 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbaa001b770 con 0x7fbab8099fd0 2023-09-30T01:59:18.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.715+0000 7fbaaffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:18.728 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.715+0000 7fbaaffff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:18.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.952+0000 7fbabd660700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:59:18.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.952+0000 7fbabd660700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:18.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.952+0000 7fbabd660700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:18.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.952+0000 7fbabd660700 1 -- 172.21.15.7:0/312689086 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbab8135820 con 0x7fbab8099fd0 2023-09-30T01:59:18.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.953+0000 7fbaaffff700 1 -- 172.21.15.7:0/312689086 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fbaa0021100 con 0x7fbab8099fd0 2023-09-30T01:59:18.953 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.953+0000 7fbaaffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:18.954 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.953+0000 7fbaaffff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:59:18.954 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:59:18.954 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:18.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.955+0000 7fbabd660700 1 -- 172.21.15.7:0/312689086 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fba980357e0 msgr2=0x7fba98037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:18.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.955+0000 7fbabd660700 1 --2- 172.21.15.7:0/312689086 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fba980357e0 0x7fba98037c90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbab8134cc0 tx=0x7fbaa8007ea0 comp rx=0 tx=0).stop 2023-09-30T01:59:18.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.955+0000 7fbabd660700 10 monclient: shutdown 2023-09-30T01:59:18.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.955+0000 7fbabd660700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:18.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.955+0000 7fbabd660700 1 -- 172.21.15.7:0/312689086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbab8099fd0 msgr2=0x7fbab8136410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:18.955 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.955+0000 7fbabd660700 1 --2- 172.21.15.7:0/312689086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbab8099fd0 0x7fbab8136410 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbaa000ec50 tx=0x7fbaa0009250 comp rx=0 tx=0).stop 2023-09-30T01:59:18.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.955+0000 7fbabd660700 1 -- 172.21.15.7:0/312689086 shutdown_connections 2023-09-30T01:59:18.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.955+0000 7fbabd660700 1 --2- 172.21.15.7:0/312689086 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fba980357e0 0x7fba98037c90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:18.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.955+0000 7fbabd660700 1 --2- 172.21.15.7:0/312689086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbab8099fd0 0x7fbab8136410 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:18.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.955+0000 7fbabd660700 1 --2- 172.21.15.7:0/312689086 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbab809c540 0x7fbab8141b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:18.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.955+0000 7fbabd660700 1 --2- 172.21.15.7:0/312689086 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fbab809ba60 0x7fbab81415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:18.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.955+0000 7fbabd660700 1 -- 172.21.15.7:0/312689086 >> 172.21.15.7:0/312689086 conn(0x7fbab80a1230 msgr2=0x7fbab809f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:18.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.955+0000 7fbabd660700 1 -- 172.21.15.7:0/312689086 shutdown_connections 2023-09-30T01:59:18.956 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:18.955+0000 7fbabd660700 1 -- 172.21.15.7:0/312689086 wait complete. 2023-09-30T01:59:21.968 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:22.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.098+0000 7f813dabb700 10 monclient: get_monmap_and_config 2023-09-30T01:59:22.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.098+0000 7f813dabb700 10 monclient: build_initial_monmap 2023-09-30T01:59:22.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.098+0000 7f813dabb700 10 monclient: monmap: 2023-09-30T01:59:22.099 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:22.099 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:22.099 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:22.099951+0000 2023-09-30T01:59:22.099 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:22.099951+0000 2023-09-30T01:59:22.099 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:22.099 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:22.099 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:22.100 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:22.100 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:22.100 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:22.100 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:22.100 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:22.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.099+0000 7f813dabb700 1 Processor -- start 2023-09-30T01:59:22.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.099+0000 7f813dabb700 1 -- start start 2023-09-30T01:59:22.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.099+0000 7f813dabb700 10 monclient: init 2023-09-30T01:59:22.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f813dabb700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:22.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f813dabb700 10 monclient: _add_conns ranks=[0,3,1,2,4] 2023-09-30T01:59:22.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f813dabb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8138098540 addr v1:172.21.15.7:6789/0 2023-09-30T01:59:22.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f813dabb700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f81381329e0 0x7f8138137400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:22.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f813dabb700 10 monclient(hunting): picked mon.noname-a con 0x7f81381329e0 addr v2:172.21.15.7:3300/0 2023-09-30T01:59:22.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f813dabb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8138137940 addr v1:172.21.15.50:6789/0 2023-09-30T01:59:22.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f813dabb700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8138069f10 con 0x7f8138098540 2023-09-30T01:59:22.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f813dabb700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8138096a50 con 0x7f8138137940 2023-09-30T01:59:22.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f813dabb700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:22.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f813dabb700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81380a0f40 con 0x7f81381329e0 2023-09-30T01:59:22.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f813dabb700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:22.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f813dabb700 10 monclient(hunting): authenticate will time out at 1451.165365s 2023-09-30T01:59:22.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f813cab9700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f8138098540 0x7f8138098910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:33312/0 (socket says 172.21.15.7:33312) 2023-09-30T01:59:22.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f8137fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f81381329e0 0x7f8138137400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:22.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f813cab9700 1 -- 172.21.15.7:0/1055445318 learned_addr learned my addr 172.21.15.7:0/1055445318 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:59:22.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f8137fff700 10 monclient(hunting): get_auth_request con 0x7f81381329e0 auth_method 0 2023-09-30T01:59:22.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f8137fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:22.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f8137fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:22.105 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f8137fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:22.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f8137fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:22.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f8137fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:22.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f8137fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:22.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f8137fff700 10 monclient(hunting): handle_auth_done global_id 4689 payload 274 2023-09-30T01:59:22.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f8137fff700 1 -- 172.21.15.7:0/1055445318 >> v1:172.21.15.50:6789/0 conn(0x7f8138137940 legacy=0x7f8138139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:22.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f8137fff700 1 -- 172.21.15.7:0/1055445318 >> v1:172.21.15.7:6789/0 conn(0x7f8138098540 legacy=0x7f8138098910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:22.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f8137fff700 10 monclient: _finish_hunting 0 2023-09-30T01:59:22.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f8137fff700 1 monclient: found mon.noname-a 2023-09-30T01:59:22.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f8137fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:22.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f8137fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:22.106 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.100+0000 7f8137fff700 1 -- 172.21.15.7:0/1055445318 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81380a05c0 con 0x7f81381329e0 2023-09-30T01:59:22.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f8137fff700 1 -- 172.21.15.7:0/1055445318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81381329e0 msgr2=0x7f8138137400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:22.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f8137fff700 1 --2- 172.21.15.7:0/1055445318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81381329e0 0x7f8138137400 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f81240098e0 tx=0x7f81240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4787783affd8c68 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:22.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f81377fe700 1 -- 172.21.15.7:0/1055445318 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812400b9f0 con 0x7f81381329e0 2023-09-30T01:59:22.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f81377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:22.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f81377fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:22.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f81377fe700 10 monclient: dump: 2023-09-30T01:59:22.114 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:22.115 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:22.115 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:22.115 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:22.115 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:22.115 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:22.115 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:22.115 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:22.115 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:22.115 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:22.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f81377fe700 10 monclient: _finish_auth 0 2023-09-30T01:59:22.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f81377fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:22.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f81377fe700 1 -- 172.21.15.7:0/1055445318 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812400bb50 con 0x7f81381329e0 2023-09-30T01:59:22.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f813dabb700 5 monclient: authenticate success, global_id 4689 2023-09-30T01:59:22.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f813dabb700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:22.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f81377fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:22.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f81377fe700 1 -- 172.21.15.7:0/1055445318 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8124011420 con 0x7f81381329e0 2023-09-30T01:59:22.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f813dabb700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:22.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f813dabb700 4 set_mon_vals no callback set 2023-09-30T01:59:22.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f81377fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:22.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f81377fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:22.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f81377fe700 10 monclient: dump: 2023-09-30T01:59:22.116 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:22.117 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:22.117 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:22.117 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:22.117 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:22.117 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:22.117 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:22.117 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:22.117 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:22.117 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:22.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f813dabb700 10 monclient: shutdown 2023-09-30T01:59:22.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f813dabb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:22.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f813dabb700 1 -- 172.21.15.7:0/1055445318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81381329e0 msgr2=0x7f8138137400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:22.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f813dabb700 1 --2- 172.21.15.7:0/1055445318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81381329e0 0x7f8138137400 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f81240098e0 tx=0x7f81240091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:22.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f813dabb700 1 -- 172.21.15.7:0/1055445318 shutdown_connections 2023-09-30T01:59:22.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f813dabb700 1 --2- 172.21.15.7:0/1055445318 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f81381329e0 0x7f8138137400 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:22.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.103+0000 7f813dabb700 1 -- 172.21.15.7:0/1055445318 >> 172.21.15.7:0/1055445318 conn(0x7f8138071830 msgr2=0x7f81380a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:22.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.104+0000 7f813dabb700 1 -- 172.21.15.7:0/1055445318 shutdown_connections 2023-09-30T01:59:22.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.104+0000 7f813dabb700 1 -- 172.21.15.7:0/1055445318 wait complete. 2023-09-30T01:59:22.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.104+0000 7f813dabb700 10 monclient: build_initial_monmap 2023-09-30T01:59:22.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.104+0000 7f813dabb700 10 monclient: monmap: 2023-09-30T01:59:22.118 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:22.118 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:22.118 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:22.119 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:22.119 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:22.119 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:22.119 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:22.119 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:22.119 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:22.119 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:22.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.104+0000 7f813dabb700 1 Processor -- start 2023-09-30T01:59:22.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813dabb700 1 -- start start 2023-09-30T01:59:22.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813dabb700 10 monclient: init 2023-09-30T01:59:22.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813dabb700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:22.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813dabb700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:59:22.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813dabb700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8138098540 0x7f813812fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:22.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813dabb700 10 monclient(hunting): picked mon.noname-c con 0x7f8138098540 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:22.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813dabb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81381329e0 0x7f8138133bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:22.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813dabb700 10 monclient(hunting): picked mon.noname-b con 0x7f81381329e0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:22.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813dabb700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8138137940 0x7f8138134110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:22.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813dabb700 10 monclient(hunting): picked mon.noname-a con 0x7f8138137940 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:22.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813dabb700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:22.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813dabb700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81381381e0 con 0x7f8138137940 2023-09-30T01:59:22.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813dabb700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:22.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813dabb700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81381327b0 con 0x7f81381329e0 2023-09-30T01:59:22.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813dabb700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:22.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813dabb700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81380a33f0 con 0x7f8138098540 2023-09-30T01:59:22.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813dabb700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:22.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813dabb700 10 monclient(hunting): authenticate will time out at 1451.170657s 2023-09-30T01:59:22.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813cab9700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8138098540 0x7f813812fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:22.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813d2ba700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8138137940 0x7f8138134110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:22.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813d2ba700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8138137940 0x7f8138134110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36852/0 (socket says 172.21.15.7:36852) 2023-09-30T01:59:22.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813d2ba700 1 -- 172.21.15.7:0/3792845691 learned_addr learned my addr 172.21.15.7:0/3792845691 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:22.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813d2ba700 10 monclient(hunting): get_auth_request con 0x7f8138137940 auth_method 0 2023-09-30T01:59:22.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813d2ba700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:22.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813d2ba700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813d2ba700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813cab9700 1 --2- 172.21.15.7:0/3792845691 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8138098540 0x7f813812fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:45004/0 (socket says 172.21.15.7:45004) 2023-09-30T01:59:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813cab9700 10 monclient(hunting): get_auth_request con 0x7f8138098540 auth_method 0 2023-09-30T01:59:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813cab9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813cab9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813cab9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813d2ba700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813d2ba700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813d2ba700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813cab9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:22.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813cab9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f813cab9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f8137fff700 1 --2- 172.21.15.7:0/3792845691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81381329e0 0x7f8138133bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f8137fff700 10 monclient(hunting): get_auth_request con 0x7f81381329e0 auth_method 0 2023-09-30T01:59:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f8137fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f8137fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.105+0000 7f8137fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.106+0000 7f813d2ba700 10 monclient(hunting): handle_auth_done global_id 4692 payload 824 2023-09-30T01:59:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.106+0000 7f813d2ba700 1 -- 172.21.15.7:0/3792845691 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8138098540 msgr2=0x7f813812fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.106+0000 7f813d2ba700 1 --2- 172.21.15.7:0/3792845691 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8138098540 0x7f813812fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.106+0000 7f813d2ba700 1 -- 172.21.15.7:0/3792845691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81381329e0 msgr2=0x7f8138133bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.106+0000 7f813d2ba700 1 --2- 172.21.15.7:0/3792845691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81381329e0 0x7f8138133bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:22.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.106+0000 7f813d2ba700 10 monclient: _finish_hunting 0 2023-09-30T01:59:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.106+0000 7f813d2ba700 1 monclient: found mon.noname-a 2023-09-30T01:59:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.106+0000 7f813d2ba700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.106+0000 7f813d2ba700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.106+0000 7f813d2ba700 1 -- 172.21.15.7:0/3792845691 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8124009420 con 0x7f8138137940 2023-09-30T01:59:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.106+0000 7f8137fff700 1 --2- 172.21.15.7:0/3792845691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81381329e0 0x7f8138133bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:59:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.106+0000 7f813cab9700 1 --2- 172.21.15.7:0/3792845691 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8138098540 0x7f813812fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f813d2ba700 1 -- 172.21.15.7:0/3792845691 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8138137940 msgr2=0x7f8138134110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f813d2ba700 1 --2- 172.21.15.7:0/3792845691 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8138137940 0x7f8138134110 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f813000c930 tx=0x7f81300026e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f8134ff9700 1 -- 172.21.15.7:0/3792845691 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8130012840 con 0x7f8138137940 2023-09-30T01:59:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f8134ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f8134ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:22.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f8134ff9700 10 monclient: dump: 2023-09-30T01:59:22.125 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:22.125 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:22.125 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:22.125 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:22.125 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:22.125 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:22.125 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:22.125 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:22.125 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:22.125 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:22.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f8134ff9700 10 monclient: _finish_auth 0 2023-09-30T01:59:22.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f8134ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:22.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f813dabb700 5 monclient: authenticate success, global_id 4692 2023-09-30T01:59:22.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f8134ff9700 1 -- 172.21.15.7:0/3792845691 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81300129c0 con 0x7f8138137940 2023-09-30T01:59:22.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f813dabb700 10 monclient: _renew_subs 2023-09-30T01:59:22.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f813dabb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:22.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f813dabb700 1 -- 172.21.15.7:0/3792845691 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813811c0f0 con 0x7f8138137940 2023-09-30T01:59:22.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f8134ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:22.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f8134ff9700 1 -- 172.21.15.7:0/3792845691 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8130012d80 con 0x7f8138137940 2023-09-30T01:59:22.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f8134ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:22.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f8134ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:22.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f8134ff9700 10 monclient: dump: 2023-09-30T01:59:22.127 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:22.127 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:22.127 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:22.127 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:22.127 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:22.127 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:22.127 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:22.127 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:22.127 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:22.127 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:22.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f8136ffd700 4 set_mon_vals no callback set 2023-09-30T01:59:22.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f813dabb700 10 monclient: _renew_subs 2023-09-30T01:59:22.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f813dabb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:22.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.108+0000 7f813dabb700 1 -- 172.21.15.7:0/3792845691 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81381363b0 con 0x7f8138137940 2023-09-30T01:59:22.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.109+0000 7f8134ff9700 1 -- 172.21.15.7:0/3792845691 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f813000a5c0 con 0x7f8138137940 2023-09-30T01:59:22.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.109+0000 7f813dabb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:22.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.109+0000 7f813dabb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:22.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.109+0000 7f813dabb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:22.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.109+0000 7f813dabb700 1 -- 172.21.15.7:0/3792845691 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813811bf50 con 0x7f8138137940 2023-09-30T01:59:22.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.109+0000 7f8134ff9700 1 --2- 172.21.15.7:0/3792845691 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f811c0357e0 0x7f811c037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:22.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.109+0000 7f8134ff9700 1 -- 172.21.15.7:0/3792845691 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f8130047a70 con 0x7f8138137940 2023-09-30T01:59:22.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.109+0000 7f813cab9700 1 --2- 172.21.15.7:0/3792845691 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f811c0357e0 0x7f811c037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:22.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.109+0000 7f813cab9700 10 monclient: get_auth_request con 0x7f811c0357e0 auth_method 0 2023-09-30T01:59:22.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.112+0000 7f813cab9700 1 --2- 172.21.15.7:0/3792845691 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f811c0357e0 0x7f811c037c90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f812c005220 tx=0x7f812c007f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:22.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.114+0000 7f8134ff9700 1 -- 172.21.15.7:0/3792845691 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f813811bf50 con 0x7f8138137940 2023-09-30T01:59:22.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.114+0000 7f8134ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:22.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.114+0000 7f8134ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:22.351 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.351+0000 7f813dabb700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:59:22.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.351+0000 7f813dabb700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:22.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.351+0000 7f813dabb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:22.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.351+0000 7f813dabb700 1 -- 172.21.15.7:0/3792845691 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f813811bf50 con 0x7f8138137940 2023-09-30T01:59:22.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.352+0000 7f8134ff9700 1 -- 172.21.15.7:0/3792845691 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f813811bf50 con 0x7f8138137940 2023-09-30T01:59:22.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.352+0000 7f8134ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:22.354 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.352+0000 7f8134ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:59:22.355 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:59:22.355 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:22.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.354+0000 7f813dabb700 1 -- 172.21.15.7:0/3792845691 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f811c0357e0 msgr2=0x7f811c037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:22.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.354+0000 7f813dabb700 1 --2- 172.21.15.7:0/3792845691 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f811c0357e0 0x7f811c037c90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f812c005220 tx=0x7f812c007f90 comp rx=0 tx=0).stop 2023-09-30T01:59:22.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.354+0000 7f813dabb700 10 monclient: shutdown 2023-09-30T01:59:22.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.354+0000 7f813dabb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:22.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.354+0000 7f813dabb700 1 -- 172.21.15.7:0/3792845691 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8138137940 msgr2=0x7f8138134110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:22.356 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.354+0000 7f813dabb700 1 --2- 172.21.15.7:0/3792845691 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8138137940 0x7f8138134110 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f813000c930 tx=0x7f81300026e0 comp rx=0 tx=0).stop 2023-09-30T01:59:22.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.354+0000 7f813dabb700 1 -- 172.21.15.7:0/3792845691 shutdown_connections 2023-09-30T01:59:22.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.354+0000 7f813dabb700 1 --2- 172.21.15.7:0/3792845691 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f811c0357e0 0x7f811c037c90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:22.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.354+0000 7f813dabb700 1 --2- 172.21.15.7:0/3792845691 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8138137940 0x7f8138134110 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:22.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.354+0000 7f813dabb700 1 --2- 172.21.15.7:0/3792845691 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8138098540 0x7f813812fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:22.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.354+0000 7f813dabb700 1 --2- 172.21.15.7:0/3792845691 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f81381329e0 0x7f8138133bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:22.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.354+0000 7f813dabb700 1 -- 172.21.15.7:0/3792845691 >> 172.21.15.7:0/3792845691 conn(0x7f8138071830 msgr2=0x7f813809ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:22.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.354+0000 7f813dabb700 1 -- 172.21.15.7:0/3792845691 shutdown_connections 2023-09-30T01:59:22.357 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:22.354+0000 7f813dabb700 1 -- 172.21.15.7:0/3792845691 wait complete. 2023-09-30T01:59:25.367 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:25.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.498+0000 7fa129a50700 10 monclient: get_monmap_and_config 2023-09-30T01:59:25.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.498+0000 7fa129a50700 10 monclient: build_initial_monmap 2023-09-30T01:59:25.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.498+0000 7fa129a50700 10 monclient: monmap: 2023-09-30T01:59:25.498 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:25.499 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:25.499 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:25.499552+0000 2023-09-30T01:59:25.499 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:25.499552+0000 2023-09-30T01:59:25.499 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:25.499 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:25.499 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:25.499 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:25.499 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:25.499 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:25.500 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:25.500 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:25.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa129a50700 1 Processor -- start 2023-09-30T01:59:25.501 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa129a50700 1 -- start start 2023-09-30T01:59:25.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa129a50700 10 monclient: init 2023-09-30T01:59:25.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa129a50700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:25.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa129a50700 10 monclient: _add_conns ranks=[0,1,2,4,3] 2023-09-30T01:59:25.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa129a50700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa124096a50 addr v1:172.21.15.7:6789/0 2023-09-30T01:59:25.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa129a50700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa124096e20 addr v1:172.21.15.50:6789/0 2023-09-30T01:59:25.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa129a50700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa124138700 0x7fa12413aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:25.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa129a50700 10 monclient(hunting): picked mon.noname-b con 0x7fa124138700 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:25.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa129a50700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa124132e60 con 0x7fa124096a50 2023-09-30T01:59:25.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa129a50700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa124099480 con 0x7fa124096e20 2023-09-30T01:59:25.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa129a50700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:25.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa129a50700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa124097bf0 con 0x7fa124138700 2023-09-30T01:59:25.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa129a50700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:25.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa129a50700 10 monclient(hunting): authenticate will time out at 1454.564809s 2023-09-30T01:59:25.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa122ffd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa124096a50 0x7fa124132750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46822/0 (socket says 172.21.15.7:46822) 2023-09-30T01:59:25.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa122ffd700 1 -- 172.21.15.7:0/4110685036 learned_addr learned my addr 172.21.15.7:0/4110685036 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:59:25.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa1237fe700 1 --2- 172.21.15.7:0/4110685036 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa124138700 0x7fa12413aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:25.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa1237fe700 10 monclient(hunting): get_auth_request con 0x7fa124138700 auth_method 0 2023-09-30T01:59:25.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa1237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:25.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa1237fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:25.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.499+0000 7fa1237fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:25.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.500+0000 7fa1237fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:25.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.500+0000 7fa1237fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:25.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.500+0000 7fa1237fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:25.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.500+0000 7fa1237fe700 10 monclient(hunting): handle_auth_done global_id 4706 payload 274 2023-09-30T01:59:25.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.500+0000 7fa1237fe700 1 -- 172.21.15.7:0/4110685036 >> v1:172.21.15.50:6789/0 conn(0x7fa124096e20 legacy=0x7fa124097240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:25.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.500+0000 7fa1237fe700 1 -- 172.21.15.7:0/4110685036 >> v1:172.21.15.7:6789/0 conn(0x7fa124096a50 legacy=0x7fa124132750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:25.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.500+0000 7fa1237fe700 10 monclient: _finish_hunting 0 2023-09-30T01:59:25.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.500+0000 7fa1237fe700 1 monclient: found mon.noname-b 2023-09-30T01:59:25.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.500+0000 7fa1237fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:25.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.500+0000 7fa1237fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:25.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.500+0000 7fa1237fe700 1 -- 172.21.15.7:0/4110685036 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1240991a0 con 0x7fa124138700 2023-09-30T01:59:25.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.502+0000 7fa1237fe700 1 -- 172.21.15.7:0/4110685036 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa124138700 msgr2=0x7fa12413aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:25.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.502+0000 7fa1237fe700 1 --2- 172.21.15.7:0/4110685036 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa124138700 0x7fa12413aae0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa11c009e90 tx=0x7fa11c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28f9f26dbf4ec97b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:25.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.502+0000 7fa121ffb700 1 -- 172.21.15.7:0/4110685036 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11c00e070 con 0x7fa124138700 2023-09-30T01:59:25.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.502+0000 7fa121ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:25.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.502+0000 7fa121ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:25.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.502+0000 7fa121ffb700 10 monclient: dump: 2023-09-30T01:59:25.510 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:25.510 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:25.510 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:25.510 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:25.510 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:25.510 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:25.511 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:25.511 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:25.511 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:25.511 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:25.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.502+0000 7fa121ffb700 10 monclient: _finish_auth 0 2023-09-30T01:59:25.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.502+0000 7fa121ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:25.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.502+0000 7fa121ffb700 1 -- 172.21.15.7:0/4110685036 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11c012430 con 0x7fa124138700 2023-09-30T01:59:25.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.502+0000 7fa129a50700 5 monclient: authenticate success, global_id 4706 2023-09-30T01:59:25.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.502+0000 7fa129a50700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:25.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.502+0000 7fa121ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:25.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.502+0000 7fa121ffb700 1 -- 172.21.15.7:0/4110685036 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11c00bea0 con 0x7fa124138700 2023-09-30T01:59:25.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.502+0000 7fa129a50700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:25.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.502+0000 7fa129a50700 4 set_mon_vals no callback set 2023-09-30T01:59:25.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.502+0000 7fa121ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:25.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.502+0000 7fa121ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:25.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.502+0000 7fa121ffb700 10 monclient: dump: 2023-09-30T01:59:25.512 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:25.512 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:25.512 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:25.512 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:25.512 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:25.512 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:25.512 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:25.512 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:25.513 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:25.513 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:25.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.503+0000 7fa129a50700 10 monclient: shutdown 2023-09-30T01:59:25.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.503+0000 7fa129a50700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:25.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.503+0000 7fa129a50700 1 -- 172.21.15.7:0/4110685036 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa124138700 msgr2=0x7fa12413aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:25.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.503+0000 7fa129a50700 1 --2- 172.21.15.7:0/4110685036 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa124138700 0x7fa12413aae0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa11c009e90 tx=0x7fa11c009780 comp rx=0 tx=0).stop 2023-09-30T01:59:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.503+0000 7fa129a50700 1 -- 172.21.15.7:0/4110685036 shutdown_connections 2023-09-30T01:59:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.503+0000 7fa129a50700 1 --2- 172.21.15.7:0/4110685036 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa124138700 0x7fa12413aae0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.503+0000 7fa129a50700 1 -- 172.21.15.7:0/4110685036 >> 172.21.15.7:0/4110685036 conn(0x7fa124070cf0 msgr2=0x7fa1240710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.503+0000 7fa129a50700 1 -- 172.21.15.7:0/4110685036 shutdown_connections 2023-09-30T01:59:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.503+0000 7fa129a50700 1 -- 172.21.15.7:0/4110685036 wait complete. 2023-09-30T01:59:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.503+0000 7fa129a50700 10 monclient: build_initial_monmap 2023-09-30T01:59:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.503+0000 7fa129a50700 10 monclient: monmap: 2023-09-30T01:59:25.514 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:25.515 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa129a50700 1 Processor -- start 2023-09-30T01:59:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa129a50700 1 -- start start 2023-09-30T01:59:25.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa129a50700 10 monclient: init 2023-09-30T01:59:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa129a50700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa129a50700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:59:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa129a50700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa124096a50 0x7fa1240a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa129a50700 10 monclient(hunting): picked mon.noname-c con 0x7fa124096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa129a50700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa124096e20 0x7fa12409c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa129a50700 10 monclient(hunting): picked mon.noname-b con 0x7fa124096e20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa129a50700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa124138700 0x7fa12409cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa129a50700 10 monclient(hunting): picked mon.noname-a con 0x7fa124138700 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa129a50700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa129a50700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1240a64a0 con 0x7fa124138700 2023-09-30T01:59:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa129a50700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa129a50700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12409fe90 con 0x7fa124096e20 2023-09-30T01:59:25.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa129a50700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa129a50700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1240a0170 con 0x7fa124096a50 2023-09-30T01:59:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa129a50700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa129a50700 10 monclient(hunting): authenticate will time out at 1454.569893s 2023-09-30T01:59:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa122ffd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa124096a50 0x7fa1240a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa122ffd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa124096a50 0x7fa1240a0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:42092/0 (socket says 172.21.15.7:42092) 2023-09-30T01:59:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.504+0000 7fa1237fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa124138700 0x7fa12409cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa122ffd700 1 -- 172.21.15.7:0/2151387551 learned_addr learned my addr 172.21.15.7:0/2151387551 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa122ffd700 10 monclient(hunting): get_auth_request con 0x7fa124096a50 auth_method 0 2023-09-30T01:59:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa122ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa122ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:25.517 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa122ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa1237fe700 10 monclient(hunting): get_auth_request con 0x7fa124138700 auth_method 0 2023-09-30T01:59:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa1237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa1237fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa1237fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa122ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa122ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa122ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa1237fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa1237fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa1237fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa1227fc700 1 --2- 172.21.15.7:0/2151387551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa124096e20 0x7fa12409c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:25.518 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa1227fc700 10 monclient(hunting): get_auth_request con 0x7fa124096e20 auth_method 0 2023-09-30T01:59:25.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa1227fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:25.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa1227fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:25.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa1227fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:25.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa122ffd700 10 monclient(hunting): handle_auth_done global_id 4709 payload 824 2023-09-30T01:59:25.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa122ffd700 1 -- 172.21.15.7:0/2151387551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa124096e20 msgr2=0x7fa12409c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:25.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa122ffd700 1 --2- 172.21.15.7:0/2151387551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa124096e20 0x7fa12409c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:25.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa122ffd700 1 -- 172.21.15.7:0/2151387551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa124138700 msgr2=0x7fa12409cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:25.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa122ffd700 1 --2- 172.21.15.7:0/2151387551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa124138700 0x7fa12409cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:25.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa122ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:59:25.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa122ffd700 1 monclient: found mon.noname-c 2023-09-30T01:59:25.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa122ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:25.519 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa122ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:59:25.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa122ffd700 1 -- 172.21.15.7:0/2151387551 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11c009a20 con 0x7fa124096a50 2023-09-30T01:59:25.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa1227fc700 1 --2- 172.21.15.7:0/2151387551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa124096e20 0x7fa12409c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:59:25.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.505+0000 7fa1237fe700 1 --2- 172.21.15.7:0/2151387551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa124138700 0x7fa12409cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:25.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa122ffd700 1 -- 172.21.15.7:0/2151387551 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa124096a50 msgr2=0x7fa1240a0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:25.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa122ffd700 1 --2- 172.21.15.7:0/2151387551 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa124096a50 0x7fa1240a0b50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa11800b5e0 tx=0x7fa118009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:25.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa10affd700 1 -- 172.21.15.7:0/2151387551 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa118011840 con 0x7fa124096a50 2023-09-30T01:59:25.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa10affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:25.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa10affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:59:25.520 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa10affd700 10 monclient: dump: 2023-09-30T01:59:25.520 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:25.520 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:25.520 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:25.521 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa10affd700 10 monclient: _finish_auth 0 2023-09-30T01:59:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa10affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa10affd700 1 -- 172.21.15.7:0/2151387551 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1180119a0 con 0x7fa124096a50 2023-09-30T01:59:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa129a50700 5 monclient: authenticate success, global_id 4709 2023-09-30T01:59:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa129a50700 10 monclient: _renew_subs 2023-09-30T01:59:25.521 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa129a50700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:25.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa129a50700 1 -- 172.21.15.7:0/2151387551 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa12409e360 con 0x7fa124096a50 2023-09-30T01:59:25.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa10affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:25.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa10affd700 1 -- 172.21.15.7:0/2151387551 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa118011d60 con 0x7fa124096a50 2023-09-30T01:59:25.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa10affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:25.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa10affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:25.522 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa10affd700 10 monclient: dump: 2023-09-30T01:59:25.522 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:25.522 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:25.522 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:25.522 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:25.522 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:25.523 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:25.523 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:25.523 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:25.523 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:25.523 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:25.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa1217fa700 4 set_mon_vals no callback set 2023-09-30T01:59:25.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa129a50700 10 monclient: _renew_subs 2023-09-30T01:59:25.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa129a50700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:25.523 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.507+0000 7fa129a50700 1 -- 172.21.15.7:0/2151387551 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1240a1090 con 0x7fa124096a50 2023-09-30T01:59:25.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.508+0000 7fa10affd700 1 -- 172.21.15.7:0/2151387551 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fa118010590 con 0x7fa124096a50 2023-09-30T01:59:25.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.508+0000 7fa129a50700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:25.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.508+0000 7fa129a50700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:25.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.508+0000 7fa129a50700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:25.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.508+0000 7fa10affd700 1 --2- 172.21.15.7:0/2151387551 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa1100357e0 0x7fa110037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:25.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.508+0000 7fa129a50700 1 -- 172.21.15.7:0/2151387551 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1241396b0 con 0x7fa124096a50 2023-09-30T01:59:25.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.508+0000 7fa10affd700 1 -- 172.21.15.7:0/2151387551 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fa118047d80 con 0x7fa124096a50 2023-09-30T01:59:25.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.509+0000 7fa1227fc700 1 --2- 172.21.15.7:0/2151387551 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa1100357e0 0x7fa110037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:25.524 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.509+0000 7fa1227fc700 10 monclient: get_auth_request con 0x7fa1100357e0 auth_method 0 2023-09-30T01:59:25.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.511+0000 7fa1227fc700 1 --2- 172.21.15.7:0/2151387551 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa1100357e0 0x7fa110037c90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa114007fd0 tx=0x7fa114009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:25.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.513+0000 7fa10affd700 1 -- 172.21.15.7:0/2151387551 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa118015070 con 0x7fa124096a50 2023-09-30T01:59:25.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.513+0000 7fa10affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:25.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.513+0000 7fa10affd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:25.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.748+0000 7fa129a50700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:59:25.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.748+0000 7fa129a50700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:25.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.748+0000 7fa129a50700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:25.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.748+0000 7fa129a50700 1 -- 172.21.15.7:0/2151387551 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa124064c10 con 0x7fa124096a50 2023-09-30T01:59:25.748 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.748+0000 7fa10affd700 1 -- 172.21.15.7:0/2151387551 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa11800f6a0 con 0x7fa124096a50 2023-09-30T01:59:25.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.748+0000 7fa10affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:25.749 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.748+0000 7fa10affd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:59:25.749 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:59:25.749 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:25.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.750+0000 7fa129a50700 1 -- 172.21.15.7:0/2151387551 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa1100357e0 msgr2=0x7fa110037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:25.750 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.750+0000 7fa129a50700 1 --2- 172.21.15.7:0/2151387551 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa1100357e0 0x7fa110037c90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa114007fd0 tx=0x7fa114009040 comp rx=0 tx=0).stop 2023-09-30T01:59:25.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.750+0000 7fa129a50700 10 monclient: shutdown 2023-09-30T01:59:25.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.750+0000 7fa129a50700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:25.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.750+0000 7fa129a50700 1 -- 172.21.15.7:0/2151387551 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa124096a50 msgr2=0x7fa1240a0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:25.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.750+0000 7fa129a50700 1 --2- 172.21.15.7:0/2151387551 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa124096a50 0x7fa1240a0b50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa11800b5e0 tx=0x7fa118009120 comp rx=0 tx=0).stop 2023-09-30T01:59:25.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.750+0000 7fa129a50700 1 -- 172.21.15.7:0/2151387551 shutdown_connections 2023-09-30T01:59:25.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.750+0000 7fa129a50700 1 --2- 172.21.15.7:0/2151387551 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa1100357e0 0x7fa110037c90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:25.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.750+0000 7fa129a50700 1 --2- 172.21.15.7:0/2151387551 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa124138700 0x7fa12409cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:25.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.750+0000 7fa129a50700 1 --2- 172.21.15.7:0/2151387551 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa124096a50 0x7fa1240a0b50 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:25.751 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.750+0000 7fa129a50700 1 --2- 172.21.15.7:0/2151387551 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa124096e20 0x7fa12409c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:25.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.750+0000 7fa129a50700 1 -- 172.21.15.7:0/2151387551 >> 172.21.15.7:0/2151387551 conn(0x7fa124070cf0 msgr2=0x7fa124139bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:25.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.750+0000 7fa129a50700 1 -- 172.21.15.7:0/2151387551 shutdown_connections 2023-09-30T01:59:25.752 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:25.750+0000 7fa129a50700 1 -- 172.21.15.7:0/2151387551 wait complete. 2023-09-30T01:59:28.763 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:28.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.893+0000 7fc02eac1700 10 monclient: get_monmap_and_config 2023-09-30T01:59:28.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.893+0000 7fc02eac1700 10 monclient: build_initial_monmap 2023-09-30T01:59:28.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.893+0000 7fc02eac1700 10 monclient: monmap: 2023-09-30T01:59:28.894 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:28.894 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:28.894 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:28.894632+0000 2023-09-30T01:59:28.894 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:28.894632+0000 2023-09-30T01:59:28.894 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:28.894 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:28.894 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:28.894 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:28.894 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:28.894 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:28.894 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:28.895 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:28.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.894+0000 7fc02eac1700 1 Processor -- start 2023-09-30T01:59:28.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.894+0000 7fc02eac1700 1 -- start start 2023-09-30T01:59:28.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.894+0000 7fc02eac1700 10 monclient: init 2023-09-30T01:59:28.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.894+0000 7fc02eac1700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:28.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.894+0000 7fc02eac1700 10 monclient: _add_conns ranks=[0,3,1,4,2] 2023-09-30T01:59:28.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.894+0000 7fc02eac1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc028096a50 addr v1:172.21.15.7:6789/0 2023-09-30T01:59:28.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.894+0000 7fc02eac1700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc028096e20 0x7fc028097240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:28.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.894+0000 7fc02eac1700 10 monclient(hunting): picked mon.noname-a con 0x7fc028096e20 addr v2:172.21.15.7:3300/0 2023-09-30T01:59:28.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.894+0000 7fc02eac1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc0281376f0 addr v1:172.21.15.50:6789/0 2023-09-30T01:59:28.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.894+0000 7fc02eac1700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc028099480 con 0x7fc028096a50 2023-09-30T01:59:28.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.894+0000 7fc02eac1700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc028097bf0 con 0x7fc0281376f0 2023-09-30T01:59:28.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.894+0000 7fc02eac1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:28.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.894+0000 7fc02eac1700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc028132ef0 con 0x7fc028096e20 2023-09-30T01:59:28.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.894+0000 7fc02eac1700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:28.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.894+0000 7fc02eac1700 10 monclient(hunting): authenticate will time out at 1457.959869s 2023-09-30T01:59:28.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.894+0000 7fc027fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fc028096e20 0x7fc028097240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:28.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.895+0000 7fc02c85d700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc028096a50 0x7fc028132750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46832/0 (socket says 172.21.15.7:46832) 2023-09-30T01:59:28.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.895+0000 7fc02c85d700 1 -- 172.21.15.7:0/2664451168 learned_addr learned my addr 172.21.15.7:0/2664451168 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:59:28.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.895+0000 7fc027fff700 10 monclient(hunting): get_auth_request con 0x7fc028096e20 auth_method 0 2023-09-30T01:59:28.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.895+0000 7fc027fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:28.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.895+0000 7fc027fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:28.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.895+0000 7fc027fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:28.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.895+0000 7fc027fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:28.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.895+0000 7fc027fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:28.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.895+0000 7fc027fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:28.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.895+0000 7fc027fff700 10 monclient(hunting): handle_auth_done global_id 4698 payload 274 2023-09-30T01:59:28.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.895+0000 7fc027fff700 1 -- 172.21.15.7:0/2664451168 >> v1:172.21.15.50:6789/0 conn(0x7fc0281376f0 legacy=0x7fc028139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:28.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.895+0000 7fc027fff700 1 -- 172.21.15.7:0/2664451168 >> v1:172.21.15.7:6789/0 conn(0x7fc028096a50 legacy=0x7fc028132750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:28.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.895+0000 7fc027fff700 10 monclient: _finish_hunting 0 2023-09-30T01:59:28.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.895+0000 7fc027fff700 1 monclient: found mon.noname-a 2023-09-30T01:59:28.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.895+0000 7fc027fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:28.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.895+0000 7fc027fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:28.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.895+0000 7fc027fff700 1 -- 172.21.15.7:0/2664451168 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0280991a0 con 0x7fc028096e20 2023-09-30T01:59:28.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.897+0000 7fc027fff700 1 -- 172.21.15.7:0/2664451168 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc028096e20 msgr2=0x7fc028097240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:28.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.897+0000 7fc027fff700 1 --2- 172.21.15.7:0/2664451168 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc028096e20 0x7fc028097240 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc0140098e0 tx=0x7fc0140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aa534b25dba48bc server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:28.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc0277fe700 1 -- 172.21.15.7:0/2664451168 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01400b9f0 con 0x7fc028096e20 2023-09-30T01:59:28.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc0277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:28.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc0277fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:28.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc0277fe700 10 monclient: dump: 2023-09-30T01:59:28.905 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:28.905 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:28.906 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:28.906 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:28.906 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:28.906 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:28.906 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:28.906 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:28.906 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:28.906 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:28.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc0277fe700 10 monclient: _finish_auth 0 2023-09-30T01:59:28.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc0277fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:28.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc0277fe700 1 -- 172.21.15.7:0/2664451168 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc01400bb50 con 0x7fc028096e20 2023-09-30T01:59:28.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc02eac1700 5 monclient: authenticate success, global_id 4698 2023-09-30T01:59:28.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc02eac1700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:28.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc0277fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:28.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc02eac1700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:28.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc0277fe700 1 -- 172.21.15.7:0/2664451168 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc014011420 con 0x7fc028096e20 2023-09-30T01:59:28.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc0277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:28.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc02eac1700 4 set_mon_vals no callback set 2023-09-30T01:59:28.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc0277fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:28.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc0277fe700 10 monclient: dump: 2023-09-30T01:59:28.907 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:28.907 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:28.907 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:28.907 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:28.908 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:28.908 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:28.908 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:28.908 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:28.908 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:28.908 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:28.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc02eac1700 10 monclient: shutdown 2023-09-30T01:59:28.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc02eac1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:28.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc02eac1700 1 -- 172.21.15.7:0/2664451168 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc028096e20 msgr2=0x7fc028097240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:28.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc02eac1700 1 --2- 172.21.15.7:0/2664451168 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc028096e20 0x7fc028097240 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc0140098e0 tx=0x7fc0140091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:28.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc02eac1700 1 -- 172.21.15.7:0/2664451168 shutdown_connections 2023-09-30T01:59:28.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc02eac1700 1 --2- 172.21.15.7:0/2664451168 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc028096e20 0x7fc028097240 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:28.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc02eac1700 1 -- 172.21.15.7:0/2664451168 >> 172.21.15.7:0/2664451168 conn(0x7fc028070cf0 msgr2=0x7fc0280710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:28.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc02eac1700 1 -- 172.21.15.7:0/2664451168 shutdown_connections 2023-09-30T01:59:28.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.898+0000 7fc02eac1700 1 -- 172.21.15.7:0/2664451168 wait complete. 2023-09-30T01:59:28.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.899+0000 7fc02eac1700 10 monclient: build_initial_monmap 2023-09-30T01:59:28.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.899+0000 7fc02eac1700 10 monclient: monmap: 2023-09-30T01:59:28.909 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:28.909 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:28.909 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:28.909 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:28.909 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:28.909 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:28.909 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:28.909 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:28.910 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.899+0000 7fc02eac1700 1 Processor -- start 2023-09-30T01:59:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.899+0000 7fc02eac1700 1 -- start start 2023-09-30T01:59:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.899+0000 7fc02eac1700 10 monclient: init 2023-09-30T01:59:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02eac1700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02eac1700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:59:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02eac1700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc028096a50 0x7fc028141ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02eac1700 10 monclient(hunting): picked mon.noname-c con 0x7fc028096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02eac1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc028096e20 0x7fc028142420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02eac1700 10 monclient(hunting): picked mon.noname-b con 0x7fc028096e20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:28.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02eac1700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0281376f0 0x7fc028142960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02eac1700 10 monclient(hunting): picked mon.noname-a con 0x7fc0281376f0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02eac1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02eac1700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0280a1020 con 0x7fc0281376f0 2023-09-30T01:59:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02eac1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02eac1700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0281334c0 con 0x7fc028096e20 2023-09-30T01:59:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02eac1700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02eac1700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc028131fa0 con 0x7fc028096a50 2023-09-30T01:59:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02eac1700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02eac1700 10 monclient(hunting): authenticate will time out at 1457.965272s 2023-09-30T01:59:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02d05e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0281376f0 0x7fc028142960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02c85d700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc028096a50 0x7fc028141ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02d05e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0281376f0 0x7fc028142960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57518/0 (socket says 172.21.15.7:57518) 2023-09-30T01:59:28.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02c85d700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc028096a50 0x7fc028141ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:42094/0 (socket says 172.21.15.7:42094) 2023-09-30T01:59:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02d05e700 1 -- 172.21.15.7:0/4204588433 learned_addr learned my addr 172.21.15.7:0/4204588433 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02d05e700 10 monclient(hunting): get_auth_request con 0x7fc0281376f0 auth_method 0 2023-09-30T01:59:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02d05e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02d05e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02d05e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02c85d700 10 monclient(hunting): get_auth_request con 0x7fc028096a50 auth_method 0 2023-09-30T01:59:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02c85d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02c85d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02c85d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02d05e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02d05e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:28.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02d05e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02c85d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02c85d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02c85d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc027fff700 1 --2- 172.21.15.7:0/4204588433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc028096e20 0x7fc028142420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc027fff700 10 monclient(hunting): get_auth_request con 0x7fc028096e20 auth_method 0 2023-09-30T01:59:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc027fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc027fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc027fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02d05e700 10 monclient(hunting): handle_auth_done global_id 4701 payload 824 2023-09-30T01:59:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02d05e700 1 -- 172.21.15.7:0/4204588433 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc028096a50 msgr2=0x7fc028141ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02d05e700 1 --2- 172.21.15.7:0/4204588433 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc028096a50 0x7fc028141ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:28.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02d05e700 1 -- 172.21.15.7:0/4204588433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc028096e20 msgr2=0x7fc028142420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:28.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02d05e700 1 --2- 172.21.15.7:0/4204588433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc028096e20 0x7fc028142420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:28.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02d05e700 10 monclient: _finish_hunting 0 2023-09-30T01:59:28.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02d05e700 1 monclient: found mon.noname-a 2023-09-30T01:59:28.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02d05e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:28.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02d05e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:28.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02d05e700 1 -- 172.21.15.7:0/4204588433 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc014009420 con 0x7fc0281376f0 2023-09-30T01:59:28.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.900+0000 7fc02c85d700 1 --2- 172.21.15.7:0/4204588433 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc028096a50 0x7fc028141ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:28.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.902+0000 7fc02d05e700 1 -- 172.21.15.7:0/4204588433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0281376f0 msgr2=0x7fc028142960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:28.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.902+0000 7fc02d05e700 1 --2- 172.21.15.7:0/4204588433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0281376f0 0x7fc028142960 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc02000c930 tx=0x7fc0200026e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:28.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.903+0000 7fc024ff9700 1 -- 172.21.15.7:0/4204588433 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc020012840 con 0x7fc0281376f0 2023-09-30T01:59:28.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.903+0000 7fc024ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:28.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.903+0000 7fc024ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:28.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.903+0000 7fc024ff9700 10 monclient: dump: 2023-09-30T01:59:28.915 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:28.915 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:28.915 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:28.915 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:28.915 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:28.915 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:28.915 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:28.915 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:28.915 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:28.915 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:28.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.903+0000 7fc024ff9700 10 monclient: _finish_auth 0 2023-09-30T01:59:28.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.903+0000 7fc024ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:28.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.903+0000 7fc024ff9700 1 -- 172.21.15.7:0/4204588433 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0200129c0 con 0x7fc0281376f0 2023-09-30T01:59:28.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.903+0000 7fc02eac1700 5 monclient: authenticate success, global_id 4701 2023-09-30T01:59:28.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.903+0000 7fc02eac1700 10 monclient: _renew_subs 2023-09-30T01:59:28.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.903+0000 7fc02eac1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:28.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.903+0000 7fc02eac1700 1 -- 172.21.15.7:0/4204588433 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0280a06f0 con 0x7fc0281376f0 2023-09-30T01:59:28.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.903+0000 7fc024ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:28.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.903+0000 7fc024ff9700 1 -- 172.21.15.7:0/4204588433 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc020012d80 con 0x7fc0281376f0 2023-09-30T01:59:28.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.903+0000 7fc024ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:28.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.903+0000 7fc024ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:28.917 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.903+0000 7fc024ff9700 10 monclient: dump: 2023-09-30T01:59:28.917 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:28.917 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:28.917 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:28.917 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:28.917 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:28.917 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:28.918 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:28.918 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:28.918 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:28.918 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:28.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.903+0000 7fc026ffd700 4 set_mon_vals no callback set 2023-09-30T01:59:28.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.903+0000 7fc02eac1700 10 monclient: _renew_subs 2023-09-30T01:59:28.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.903+0000 7fc02eac1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:28.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.903+0000 7fc02eac1700 1 -- 172.21.15.7:0/4204588433 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0280a1670 con 0x7fc0281376f0 2023-09-30T01:59:28.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.903+0000 7fc024ff9700 1 -- 172.21.15.7:0/4204588433 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fc02000a5c0 con 0x7fc0281376f0 2023-09-30T01:59:28.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.904+0000 7fc02eac1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:28.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.904+0000 7fc02eac1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:28.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.904+0000 7fc02eac1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:28.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.904+0000 7fc02eac1700 1 -- 172.21.15.7:0/4204588433 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0280a0af0 con 0x7fc0281376f0 2023-09-30T01:59:28.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.904+0000 7fc024ff9700 1 --2- 172.21.15.7:0/4204588433 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc00c035830 0x7fc00c037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:28.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.904+0000 7fc024ff9700 1 -- 172.21.15.7:0/4204588433 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fc020047d20 con 0x7fc0281376f0 2023-09-30T01:59:28.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.904+0000 7fc02c85d700 1 --2- 172.21.15.7:0/4204588433 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc00c035830 0x7fc00c037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:28.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.904+0000 7fc02c85d700 10 monclient: get_auth_request con 0x7fc00c035830 auth_method 0 2023-09-30T01:59:28.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.907+0000 7fc02c85d700 1 --2- 172.21.15.7:0/4204588433 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc00c035830 0x7fc00c037ce0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc01c005220 tx=0x7fc01c007f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:28.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.908+0000 7fc024ff9700 1 -- 172.21.15.7:0/4204588433 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc0200108d0 con 0x7fc0281376f0 2023-09-30T01:59:28.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.908+0000 7fc024ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:28.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:28.908+0000 7fc024ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:29.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:29.149+0000 7fc02eac1700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:59:29.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:29.149+0000 7fc02eac1700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:29.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:29.149+0000 7fc02eac1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:29.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:29.149+0000 7fc02eac1700 1 -- 172.21.15.7:0/4204588433 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc0280a5ea0 con 0x7fc0281376f0 2023-09-30T01:59:29.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:29.150+0000 7fc024ff9700 1 -- 172.21.15.7:0/4204588433 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc020011380 con 0x7fc0281376f0 2023-09-30T01:59:29.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:29.150+0000 7fc024ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:29.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:29.150+0000 7fc024ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:59:29.151 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:59:29.151 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:29.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:29.152+0000 7fc02eac1700 1 -- 172.21.15.7:0/4204588433 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc00c035830 msgr2=0x7fc00c037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:29.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:29.152+0000 7fc02eac1700 1 --2- 172.21.15.7:0/4204588433 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc00c035830 0x7fc00c037ce0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc01c005220 tx=0x7fc01c007f90 comp rx=0 tx=0).stop 2023-09-30T01:59:29.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:29.152+0000 7fc02eac1700 10 monclient: shutdown 2023-09-30T01:59:29.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:29.152+0000 7fc02eac1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:29.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:29.152+0000 7fc02eac1700 1 -- 172.21.15.7:0/4204588433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0281376f0 msgr2=0x7fc028142960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:29.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:29.152+0000 7fc02eac1700 1 --2- 172.21.15.7:0/4204588433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0281376f0 0x7fc028142960 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc02000c930 tx=0x7fc0200026e0 comp rx=0 tx=0).stop 2023-09-30T01:59:29.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:29.152+0000 7fc02eac1700 1 -- 172.21.15.7:0/4204588433 shutdown_connections 2023-09-30T01:59:29.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:29.152+0000 7fc02eac1700 1 --2- 172.21.15.7:0/4204588433 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc00c035830 0x7fc00c037ce0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:29.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:29.152+0000 7fc02eac1700 1 --2- 172.21.15.7:0/4204588433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0281376f0 0x7fc028142960 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:29.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:29.152+0000 7fc02eac1700 1 --2- 172.21.15.7:0/4204588433 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc028096a50 0x7fc028141ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:29.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:29.152+0000 7fc02eac1700 1 --2- 172.21.15.7:0/4204588433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc028096e20 0x7fc028142420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:29.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:29.152+0000 7fc02eac1700 1 -- 172.21.15.7:0/4204588433 >> 172.21.15.7:0/4204588433 conn(0x7fc028070cf0 msgr2=0x7fc02812f240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:29.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:29.152+0000 7fc02eac1700 1 -- 172.21.15.7:0/4204588433 shutdown_connections 2023-09-30T01:59:29.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:29.152+0000 7fc02eac1700 1 -- 172.21.15.7:0/4204588433 wait complete. 2023-09-30T01:59:32.165 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:32.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.297+0000 7f1335996700 10 monclient: get_monmap_and_config 2023-09-30T01:59:32.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.297+0000 7f1335996700 10 monclient: build_initial_monmap 2023-09-30T01:59:32.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.297+0000 7f1335996700 10 monclient: monmap: 2023-09-30T01:59:32.300 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:32.300 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:32.300 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:32.298082+0000 2023-09-30T01:59:32.300 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:32.298082+0000 2023-09-30T01:59:32.300 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:32.300 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:32.301 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:32.301 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:32.301 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:32.301 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:32.301 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:32.301 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:32.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.297+0000 7f1335996700 1 Processor -- start 2023-09-30T01:59:32.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.297+0000 7f1335996700 1 -- start start 2023-09-30T01:59:32.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.297+0000 7f1335996700 10 monclient: init 2023-09-30T01:59:32.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f1335996700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:32.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f1335996700 10 monclient: _add_conns ranks=[1,4,3,0,2] 2023-09-30T01:59:32.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f1335996700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1330096a50 addr v1:172.21.15.50:6789/0 2023-09-30T01:59:32.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f1335996700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f1330096e20 0x7f1330097220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:32.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f1335996700 10 monclient(hunting): picked mon.noname-c con 0x7f1330096e20 addr v2:172.21.15.50:3300/0 2023-09-30T01:59:32.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f1335996700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1330137710 0x7f1330139af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:32.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f1335996700 10 monclient(hunting): picked mon.noname-a con 0x7f1330137710 addr v2:172.21.15.7:3300/0 2023-09-30T01:59:32.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f1335996700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1330132d90 con 0x7f1330096a50 2023-09-30T01:59:32.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f1335996700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:32.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f1335996700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1330097b40 con 0x7f1330137710 2023-09-30T01:59:32.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f1335996700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:32.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f1335996700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1330099160 con 0x7f1330096e20 2023-09-30T01:59:32.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f1335996700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:32.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f1335996700 10 monclient(hunting): authenticate will time out at 1461.363184s 2023-09-30T01:59:32.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1330137710 0x7f1330139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:32.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132f7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f1330137710 0x7f1330139af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57530/0 (socket says 172.21.15.7:57530) 2023-09-30T01:59:32.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132f7fe700 1 -- 172.21.15.7:0/2615289470 learned_addr learned my addr 172.21.15.7:0/2615289470 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:32.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132f7fe700 10 monclient(hunting): get_auth_request con 0x7f1330137710 auth_method 0 2023-09-30T01:59:32.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:32.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:32.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:32.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:32.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:32.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132e7fc700 1 --2- 172.21.15.7:0/2615289470 >> v2:172.21.15.50:3300/0 conn(0x7f1330096e20 0x7f1330097220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:32.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:32.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132e7fc700 10 monclient(hunting): get_auth_request con 0x7f1330096e20 auth_method 0 2023-09-30T01:59:32.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:32.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:32.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:32.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:32.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:32.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:32.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132f7fe700 10 monclient(hunting): handle_auth_done global_id 4704 payload 274 2023-09-30T01:59:32.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132f7fe700 1 -- 172.21.15.7:0/2615289470 >> v2:172.21.15.50:3300/0 conn(0x7f1330096e20 msgr2=0x7f1330097220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:32.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132f7fe700 1 --2- 172.21.15.7:0/2615289470 >> v2:172.21.15.50:3300/0 conn(0x7f1330096e20 0x7f1330097220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:32.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132f7fe700 1 -- 172.21.15.7:0/2615289470 >> v1:172.21.15.50:6789/0 conn(0x7f1330096a50 legacy=0x7f1330132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:32.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132f7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:59:32.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132f7fe700 1 monclient: found mon.noname-a 2023-09-30T01:59:32.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:32.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:32.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.298+0000 7f132f7fe700 1 -- 172.21.15.7:0/2615289470 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1330099720 con 0x7f1330137710 2023-09-30T01:59:32.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.300+0000 7f132f7fe700 1 -- 172.21.15.7:0/2615289470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1330137710 msgr2=0x7f1330139af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:32.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.300+0000 7f132f7fe700 1 --2- 172.21.15.7:0/2615289470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1330137710 0x7f1330139af0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f13300a5740 tx=0x7f1328009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30239eeda4a02dd4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:32.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f132dffb700 1 -- 172.21.15.7:0/2615289470 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1328009e70 con 0x7f1330137710 2023-09-30T01:59:32.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f132dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:32.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f132dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:32.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f132dffb700 10 monclient: dump: 2023-09-30T01:59:32.313 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:32.313 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:32.313 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:32.313 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:32.313 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:32.313 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:32.313 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:32.313 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:32.313 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:32.313 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:32.313 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f132dffb700 10 monclient: _finish_auth 0 2023-09-30T01:59:32.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f132dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:32.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f132dffb700 1 -- 172.21.15.7:0/2615289470 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132800ba30 con 0x7f1330137710 2023-09-30T01:59:32.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f1335996700 5 monclient: authenticate success, global_id 4704 2023-09-30T01:59:32.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f1335996700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:32.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f132dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:32.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f1335996700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:32.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f132dffb700 1 -- 172.21.15.7:0/2615289470 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132800bdf0 con 0x7f1330137710 2023-09-30T01:59:32.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f1335996700 4 set_mon_vals no callback set 2023-09-30T01:59:32.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f132dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:32.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f132dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:32.314 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f132dffb700 10 monclient: dump: 2023-09-30T01:59:32.314 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:32.315 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:32.315 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:32.315 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:32.315 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:32.315 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:32.315 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:32.315 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:32.315 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:32.315 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:32.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f1335996700 10 monclient: shutdown 2023-09-30T01:59:32.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f1335996700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:32.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f1335996700 1 -- 172.21.15.7:0/2615289470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1330137710 msgr2=0x7f1330139af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:32.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f1335996700 1 --2- 172.21.15.7:0/2615289470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1330137710 0x7f1330139af0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f13300a5740 tx=0x7f1328009250 comp rx=0 tx=0).stop 2023-09-30T01:59:32.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f1335996700 1 -- 172.21.15.7:0/2615289470 shutdown_connections 2023-09-30T01:59:32.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f1335996700 1 --2- 172.21.15.7:0/2615289470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1330137710 0x7f1330139af0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:32.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f1335996700 1 --2- 172.21.15.7:0/2615289470 >> v2:172.21.15.50:3300/0 conn(0x7f1330096e20 0x7f1330097220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:32.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f1335996700 1 -- 172.21.15.7:0/2615289470 >> 172.21.15.7:0/2615289470 conn(0x7f1330070cf0 msgr2=0x7f13300710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:32.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f1335996700 1 -- 172.21.15.7:0/2615289470 shutdown_connections 2023-09-30T01:59:32.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.301+0000 7f1335996700 1 -- 172.21.15.7:0/2615289470 wait complete. 2023-09-30T01:59:32.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.302+0000 7f1335996700 10 monclient: build_initial_monmap 2023-09-30T01:59:32.316 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.302+0000 7f1335996700 10 monclient: monmap: 2023-09-30T01:59:32.317 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:32.317 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:32.317 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:32.317 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:32.317 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:32.317 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:32.317 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:32.317 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:32.317 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:32.317 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:32.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.302+0000 7f1335996700 1 Processor -- start 2023-09-30T01:59:32.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.302+0000 7f1335996700 1 -- start start 2023-09-30T01:59:32.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.302+0000 7f1335996700 10 monclient: init 2023-09-30T01:59:32.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f1335996700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:32.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f1335996700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:59:32.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f1335996700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1330096a50 0x7f133012d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:32.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f1335996700 10 monclient(hunting): picked mon.noname-c con 0x7f1330096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:32.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f1335996700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1330096e20 0x7f133012d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:32.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f1335996700 10 monclient(hunting): picked mon.noname-b con 0x7f1330096e20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:32.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f1335996700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1330137710 0x7f133009bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:32.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f1335996700 10 monclient(hunting): picked mon.noname-a con 0x7f1330137710 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:32.318 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f1335996700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f1335996700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133009f5c0 con 0x7f1330137710 2023-09-30T01:59:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f1335996700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f1335996700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1330131ad0 con 0x7f1330096e20 2023-09-30T01:59:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f1335996700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f1335996700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13301334e0 con 0x7f1330096a50 2023-09-30T01:59:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f1335996700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f1335996700 10 monclient(hunting): authenticate will time out at 1461.368292s 2023-09-30T01:59:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132effd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1330096a50 0x7f133012d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132f7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1330137710 0x7f133009bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132effd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1330096a50 0x7f133012d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:42122/0 (socket says 172.21.15.7:42122) 2023-09-30T01:59:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132effd700 1 -- 172.21.15.7:0/217791450 learned_addr learned my addr 172.21.15.7:0/217791450 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:32.319 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132effd700 10 monclient(hunting): get_auth_request con 0x7f1330096a50 auth_method 0 2023-09-30T01:59:32.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:32.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:32.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:32.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132f7fe700 10 monclient(hunting): get_auth_request con 0x7f1330137710 auth_method 0 2023-09-30T01:59:32.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:32.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:32.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:32.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:32.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:32.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:32.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:32.320 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132e7fc700 1 --2- 172.21.15.7:0/217791450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1330096e20 0x7f133012d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132f7fe700 10 monclient(hunting): handle_auth_done global_id 4707 payload 824 2023-09-30T01:59:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132f7fe700 1 -- 172.21.15.7:0/217791450 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1330096a50 msgr2=0x7f133012d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132f7fe700 1 --2- 172.21.15.7:0/217791450 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1330096a50 0x7f133012d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132f7fe700 1 -- 172.21.15.7:0/217791450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1330096e20 msgr2=0x7f133012d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132f7fe700 1 --2- 172.21.15.7:0/217791450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1330096e20 0x7f133012d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132f7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:59:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132f7fe700 1 monclient: found mon.noname-a 2023-09-30T01:59:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:32.321 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.303+0000 7f132f7fe700 1 -- 172.21.15.7:0/217791450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13280094f0 con 0x7f1330137710 2023-09-30T01:59:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.305+0000 7f132f7fe700 1 -- 172.21.15.7:0/217791450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1330137710 msgr2=0x7f133009bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.305+0000 7f132f7fe700 1 --2- 172.21.15.7:0/217791450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1330137710 0x7f133009bc50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1328015000 tx=0x7f1328016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.306+0000 7f1326ffd700 1 -- 172.21.15.7:0/217791450 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1328011b90 con 0x7f1330137710 2023-09-30T01:59:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.306+0000 7f1326ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.306+0000 7f1326ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.306+0000 7f1326ffd700 10 monclient: dump: 2023-09-30T01:59:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:32.322 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:32.323 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.306+0000 7f1326ffd700 10 monclient: _finish_auth 0 2023-09-30T01:59:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.306+0000 7f1326ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.306+0000 7f1326ffd700 1 -- 172.21.15.7:0/217791450 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1328020070 con 0x7f1330137710 2023-09-30T01:59:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.306+0000 7f1335996700 5 monclient: authenticate success, global_id 4707 2023-09-30T01:59:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.306+0000 7f1326ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.306+0000 7f1326ffd700 1 -- 172.21.15.7:0/217791450 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132801b450 con 0x7f1330137710 2023-09-30T01:59:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.306+0000 7f1335996700 10 monclient: _renew_subs 2023-09-30T01:59:32.323 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.306+0000 7f1335996700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.306+0000 7f1335996700 1 -- 172.21.15.7:0/217791450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f133009e450 con 0x7f1330137710 2023-09-30T01:59:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.306+0000 7f1326ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.306+0000 7f132d7fa700 4 set_mon_vals no callback set 2023-09-30T01:59:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.306+0000 7f1326ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.306+0000 7f1326ffd700 10 monclient: dump: 2023-09-30T01:59:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:32.324 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:32.325 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:32.325 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:32.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.306+0000 7f1335996700 10 monclient: _renew_subs 2023-09-30T01:59:32.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.306+0000 7f1335996700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:32.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.306+0000 7f1335996700 1 -- 172.21.15.7:0/217791450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13300a0570 con 0x7f1330137710 2023-09-30T01:59:32.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.306+0000 7f1326ffd700 1 -- 172.21.15.7:0/217791450 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f1328022030 con 0x7f1330137710 2023-09-30T01:59:32.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.307+0000 7f1335996700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:32.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.307+0000 7f1335996700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:32.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.307+0000 7f1335996700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:32.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.307+0000 7f1335996700 1 -- 172.21.15.7:0/217791450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13300a2d20 con 0x7f1330137710 2023-09-30T01:59:32.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.307+0000 7f1326ffd700 1 --2- 172.21.15.7:0/217791450 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f13100357e0 0x7f1310037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:32.325 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.307+0000 7f1326ffd700 1 -- 172.21.15.7:0/217791450 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f1328049af0 con 0x7f1330137710 2023-09-30T01:59:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.307+0000 7f132effd700 1 --2- 172.21.15.7:0/217791450 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f13100357e0 0x7f1310037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.307+0000 7f132effd700 10 monclient: get_auth_request con 0x7f13100357e0 auth_method 0 2023-09-30T01:59:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.310+0000 7f132effd700 1 --2- 172.21.15.7:0/217791450 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f13100357e0 0x7f1310037c90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1320007fd0 tx=0x7f1320002760 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.311+0000 7f1326ffd700 1 -- 172.21.15.7:0/217791450 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f13280249c0 con 0x7f1330137710 2023-09-30T01:59:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.311+0000 7f1326ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:32.326 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.311+0000 7f1326ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:32.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.550+0000 7f1335996700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:59:32.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.550+0000 7f1335996700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:32.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.550+0000 7f1335996700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:32.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.550+0000 7f1335996700 1 -- 172.21.15.7:0/217791450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f13300a2d20 con 0x7f1330137710 2023-09-30T01:59:32.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.550+0000 7f1326ffd700 1 -- 172.21.15.7:0/217791450 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f13280193f0 con 0x7f1330137710 2023-09-30T01:59:32.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.550+0000 7f1326ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:32.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.550+0000 7f1326ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:59:32.551 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:59:32.551 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:32.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.552+0000 7f1335996700 1 -- 172.21.15.7:0/217791450 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f13100357e0 msgr2=0x7f1310037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:32.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.552+0000 7f1335996700 1 --2- 172.21.15.7:0/217791450 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f13100357e0 0x7f1310037c90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1320007fd0 tx=0x7f1320002760 comp rx=0 tx=0).stop 2023-09-30T01:59:32.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.552+0000 7f1335996700 10 monclient: shutdown 2023-09-30T01:59:32.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.552+0000 7f1335996700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:32.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.552+0000 7f1335996700 1 -- 172.21.15.7:0/217791450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1330137710 msgr2=0x7f133009bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:32.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.552+0000 7f1335996700 1 --2- 172.21.15.7:0/217791450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1330137710 0x7f133009bc50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1328015000 tx=0x7f1328016000 comp rx=0 tx=0).stop 2023-09-30T01:59:32.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.553+0000 7f1335996700 1 -- 172.21.15.7:0/217791450 shutdown_connections 2023-09-30T01:59:32.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.553+0000 7f1335996700 1 --2- 172.21.15.7:0/217791450 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f13100357e0 0x7f1310037c90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:32.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.553+0000 7f1335996700 1 --2- 172.21.15.7:0/217791450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1330137710 0x7f133009bc50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:32.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.553+0000 7f1335996700 1 --2- 172.21.15.7:0/217791450 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f1330096a50 0x7f133012d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:32.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.553+0000 7f1335996700 1 --2- 172.21.15.7:0/217791450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1330096e20 0x7f133012d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:32.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.553+0000 7f1335996700 1 -- 172.21.15.7:0/217791450 >> 172.21.15.7:0/217791450 conn(0x7f1330070cf0 msgr2=0x7f1330131ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:32.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.553+0000 7f1335996700 1 -- 172.21.15.7:0/217791450 shutdown_connections 2023-09-30T01:59:32.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:32.553+0000 7f1335996700 1 -- 172.21.15.7:0/217791450 wait complete. 2023-09-30T01:59:34.355 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.354+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:59:34.355 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.354+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=11,kv=9,logm=68} crc {auth=3454872041,config=3140514009,health=3387429721,kv=1057557306,logm=3372656591}) 2023-09-30T01:59:34.355 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.354+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=12,kv=10,logm=65} crc {auth=348207949,config=3140514009,health=1307886691,kv=2069498558,logm=3854140668}) 2023-09-30T01:59:34.355 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.354+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:59:34.355 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.354+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=11,kv=9,logm=68} crc {auth=3454872041,config=3140514009,health=3387429721,kv=1057557306,logm=3372656591}) 2023-09-30T01:59:34.355 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.354+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=12,kv=9,logm=67} crc {auth=2820046732,config=3140514009,health=1307886691,kv=2069498558,logm=1577021982}) 2023-09-30T01:59:34.356 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.355+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:59:34.356 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.355+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4208924165}) 2023-09-30T01:59:34.357 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.355+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=77045201}) 2023-09-30T01:59:34.357 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.355+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:59:34.357 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.355+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4208924165}) 2023-09-30T01:59:34.357 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.355+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3122840652}) 2023-09-30T01:59:34.357 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.356+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:59:34.357 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.356+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3107295902}) 2023-09-30T01:59:34.357 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.356+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3991288042}) 2023-09-30T01:59:34.357 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.356+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:59:34.357 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.356+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3107295902}) 2023-09-30T01:59:34.357 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.356+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1003553938}) 2023-09-30T01:59:34.359 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.357+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:59:34.359 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.357+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=427791831}) 2023-09-30T01:59:34.359 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.357+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1380518683}) 2023-09-30T01:59:34.359 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.357+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:59:34.360 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.357+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=427791831}) 2023-09-30T01:59:34.360 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.357+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=923401874}) 2023-09-30T01:59:34.360 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.358+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:59:34.360 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.358+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283091370}) 2023-09-30T01:59:34.360 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.358+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2809112514}) 2023-09-30T01:59:34.360 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.358+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:59:34.360 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.358+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283091370}) 2023-09-30T01:59:34.360 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.358+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1029854198}) 2023-09-30T01:59:34.360 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.359+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:59:34.360 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.359+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=343992712}) 2023-09-30T01:59:34.360 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.359+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1665237252}) 2023-09-30T01:59:34.360 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.359+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:59:34.360 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.359+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=343992712}) 2023-09-30T01:59:34.361 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.359+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=484665822}) 2023-09-30T01:59:34.361 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.360+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:59:34.361 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.360+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=35,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=29} crc {logm=61738361,mdsmap=1531151339,mgr=4227378002,mgr_command_descs=3077679017,mgr_metadata=296514687,mgrstat=3320878577,mon_config_key=875215273,monmap=3823033386,osd_metadata=1821053098,osd_pg_creating=534015559,osdmap=2669565793}) 2023-09-30T01:59:34.362 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.360+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=36,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=30} crc {logm=3605227359,mdsmap=1531151339,mgr=4227378002,mgr_command_descs=3077679017,mgr_metadata=296514687,mgrstat=3320878577,mon_config_key=639161190,monmap=3823033386,osd_metadata=365889904,osd_pg_creating=534015559,osdmap=4200720525}) 2023-09-30T01:59:34.362 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.360+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T01:59:34.362 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.360+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=35,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=29} crc {logm=61738361,mdsmap=1531151339,mgr=4227378002,mgr_command_descs=3077679017,mgr_metadata=296514687,mgrstat=3320878577,mon_config_key=875215273,monmap=3823033386,osd_metadata=1821053098,osd_pg_creating=534015559,osdmap=2669565793}) 2023-09-30T01:59:34.362 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:34.360+0000 7fb5aecec700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=35,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=30} crc {logm=3636716849,mdsmap=1531151339,mgr=4227378002,mgr_command_descs=3077679017,mgr_metadata=296514687,mgrstat=3766491209,mon_config_key=875215273,monmap=3823033386,osd_metadata=2808706981,osd_pg_creating=534015559,osdmap=3724289195}) 2023-09-30T01:59:35.565 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:35.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.695+0000 7f9a2959f700 10 monclient: get_monmap_and_config 2023-09-30T01:59:35.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.695+0000 7f9a2959f700 10 monclient: build_initial_monmap 2023-09-30T01:59:35.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.695+0000 7f9a2959f700 10 monclient: monmap: 2023-09-30T01:59:35.695 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:35.695 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:35.696 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:35.696537+0000 2023-09-30T01:59:35.696 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:35.696537+0000 2023-09-30T01:59:35.696 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:35.696 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:35.696 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:35.696 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:35.696 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:35.696 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:35.696 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:35.696 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:35.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.696+0000 7f9a2959f700 1 Processor -- start 2023-09-30T01:59:35.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.696+0000 7f9a2959f700 1 -- start start 2023-09-30T01:59:35.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.696+0000 7f9a2959f700 10 monclient: init 2023-09-30T01:59:35.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.696+0000 7f9a2959f700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:35.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.696+0000 7f9a2959f700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2023-09-30T01:59:35.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.696+0000 7f9a2959f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9a24098540 addr v1:172.21.15.50:6789/0 2023-09-30T01:59:35.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.696+0000 7f9a2959f700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a241329e0 0x7f9a24137400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:35.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.696+0000 7f9a2959f700 10 monclient(hunting): picked mon.noname-b con 0x7f9a241329e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:35.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.696+0000 7f9a2959f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a24137940 addr v1:172.21.15.7:6789/0 2023-09-30T01:59:35.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.696+0000 7f9a2959f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a24069f10 con 0x7f9a24137940 2023-09-30T01:59:35.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.696+0000 7f9a2959f700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a24096a50 con 0x7f9a24098540 2023-09-30T01:59:35.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.696+0000 7f9a2959f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:35.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.696+0000 7f9a2959f700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a240a0f40 con 0x7f9a241329e0 2023-09-30T01:59:35.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.696+0000 7f9a2959f700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:35.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.696+0000 7f9a2959f700 10 monclient(hunting): authenticate will time out at 1464.761944s 2023-09-30T01:59:35.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.697+0000 7f9a237fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a241329e0 0x7f9a24137400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:35.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.697+0000 7f9a28d9e700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9a24137940 0x7f9a24139d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55700/0 (socket says 172.21.15.7:55700) 2023-09-30T01:59:35.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.697+0000 7f9a237fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a241329e0 0x7f9a24137400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:34470/0 (socket says 172.21.15.7:34470) 2023-09-30T01:59:35.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.697+0000 7f9a28d9e700 1 -- 172.21.15.7:0/2204920713 learned_addr learned my addr 172.21.15.7:0/2204920713 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:59:35.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.697+0000 7f9a237fe700 10 monclient(hunting): get_auth_request con 0x7f9a241329e0 auth_method 0 2023-09-30T01:59:35.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.697+0000 7f9a237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:35.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.697+0000 7f9a237fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:35.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.697+0000 7f9a237fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:35.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.697+0000 7f9a237fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:35.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.697+0000 7f9a237fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:35.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.697+0000 7f9a237fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:35.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.697+0000 7f9a237fe700 10 monclient(hunting): handle_auth_done global_id 4718 payload 274 2023-09-30T01:59:35.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.697+0000 7f9a237fe700 1 -- 172.21.15.7:0/2204920713 >> v1:172.21.15.50:6789/0 conn(0x7f9a24098540 legacy=0x7f9a24098910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:35.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.697+0000 7f9a237fe700 1 -- 172.21.15.7:0/2204920713 >> v1:172.21.15.7:6789/0 conn(0x7f9a24137940 legacy=0x7f9a24139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:35.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.697+0000 7f9a237fe700 10 monclient: _finish_hunting 0 2023-09-30T01:59:35.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.697+0000 7f9a237fe700 1 monclient: found mon.noname-b 2023-09-30T01:59:35.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.697+0000 7f9a237fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:35.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.697+0000 7f9a237fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:35.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.697+0000 7f9a237fe700 1 -- 172.21.15.7:0/2204920713 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a240a05c0 con 0x7f9a241329e0 2023-09-30T01:59:35.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.699+0000 7f9a237fe700 1 -- 172.21.15.7:0/2204920713 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a241329e0 msgr2=0x7f9a24137400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:35.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.699+0000 7f9a237fe700 1 --2- 172.21.15.7:0/2204920713 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a241329e0 0x7f9a24137400 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9a14009940 tx=0x7f9a140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53c65bc8384f7dfd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:35.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.699+0000 7f9a22ffd700 1 -- 172.21.15.7:0/2204920713 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1400b9f0 con 0x7f9a241329e0 2023-09-30T01:59:35.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.699+0000 7f9a22ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:35.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.699+0000 7f9a22ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:35.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.699+0000 7f9a22ffd700 10 monclient: dump: 2023-09-30T01:59:35.709 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:35.709 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:35.709 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:35.709 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:35.709 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:35.709 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:35.709 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:35.709 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:35.709 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:35.709 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:35.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.699+0000 7f9a22ffd700 10 monclient: _finish_auth 0 2023-09-30T01:59:35.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.699+0000 7f9a22ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:35.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.699+0000 7f9a2959f700 5 monclient: authenticate success, global_id 4718 2023-09-30T01:59:35.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.699+0000 7f9a2959f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:35.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.699+0000 7f9a22ffd700 1 -- 172.21.15.7:0/2204920713 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a1400bb50 con 0x7f9a241329e0 2023-09-30T01:59:35.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.699+0000 7f9a22ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:35.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.699+0000 7f9a2959f700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:35.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.699+0000 7f9a2959f700 4 set_mon_vals no callback set 2023-09-30T01:59:35.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.699+0000 7f9a22ffd700 1 -- 172.21.15.7:0/2204920713 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a14011420 con 0x7f9a241329e0 2023-09-30T01:59:35.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.699+0000 7f9a22ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:35.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.699+0000 7f9a22ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:35.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.699+0000 7f9a22ffd700 10 monclient: dump: 2023-09-30T01:59:35.711 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:35.711 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:35.712 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:35.712 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:35.712 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:35.712 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:35.712 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:35.712 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:35.712 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:35.712 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:35.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.700+0000 7f9a2959f700 10 monclient: shutdown 2023-09-30T01:59:35.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.700+0000 7f9a2959f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:35.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.700+0000 7f9a2959f700 1 -- 172.21.15.7:0/2204920713 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a241329e0 msgr2=0x7f9a24137400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:35.712 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.700+0000 7f9a2959f700 1 --2- 172.21.15.7:0/2204920713 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a241329e0 0x7f9a24137400 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9a14009940 tx=0x7f9a140091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:35.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.700+0000 7f9a2959f700 1 -- 172.21.15.7:0/2204920713 shutdown_connections 2023-09-30T01:59:35.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.700+0000 7f9a2959f700 1 --2- 172.21.15.7:0/2204920713 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a241329e0 0x7f9a24137400 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:35.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.700+0000 7f9a2959f700 1 -- 172.21.15.7:0/2204920713 >> 172.21.15.7:0/2204920713 conn(0x7f9a24071830 msgr2=0x7f9a240a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:35.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.700+0000 7f9a2959f700 1 -- 172.21.15.7:0/2204920713 shutdown_connections 2023-09-30T01:59:35.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.700+0000 7f9a2959f700 1 -- 172.21.15.7:0/2204920713 wait complete. 2023-09-30T01:59:35.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.700+0000 7f9a2959f700 10 monclient: build_initial_monmap 2023-09-30T01:59:35.713 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.700+0000 7f9a2959f700 10 monclient: monmap: 2023-09-30T01:59:35.713 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:35.713 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:35.713 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:35.713 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:35.713 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:35.714 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:35.714 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:35.714 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:35.714 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:35.714 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:35.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a2959f700 1 Processor -- start 2023-09-30T01:59:35.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a2959f700 1 -- start start 2023-09-30T01:59:35.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a2959f700 10 monclient: init 2023-09-30T01:59:35.714 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a2959f700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:35.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a2959f700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:59:35.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a2959f700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a24098540 0x7f9a24133150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:35.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a2959f700 10 monclient(hunting): picked mon.noname-c con 0x7f9a24098540 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:35.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a2959f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a241329e0 0x7f9a24133690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:35.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a2959f700 10 monclient(hunting): picked mon.noname-a con 0x7f9a241329e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:35.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a2959f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a24137940 0x7f9a24133bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:35.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a2959f700 10 monclient(hunting): picked mon.noname-b con 0x7f9a24137940 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:35.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a2959f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:35.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a2959f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a24136110 con 0x7f9a241329e0 2023-09-30T01:59:35.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a2959f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:35.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a2959f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a241327b0 con 0x7f9a24137940 2023-09-30T01:59:35.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a2959f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:35.715 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a2959f700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2412f150 con 0x7f9a24098540 2023-09-30T01:59:35.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a2959f700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:35.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a2959f700 10 monclient(hunting): authenticate will time out at 1464.766859s 2023-09-30T01:59:35.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a237fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a241329e0 0x7f9a24133690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:35.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a23fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a24098540 0x7f9a24133150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:35.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a237fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a241329e0 0x7f9a24133690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46742/0 (socket says 172.21.15.7:46742) 2023-09-30T01:59:35.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a237fe700 1 -- 172.21.15.7:0/657461129 learned_addr learned my addr 172.21.15.7:0/657461129 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:35.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a237fe700 10 monclient(hunting): get_auth_request con 0x7f9a241329e0 auth_method 0 2023-09-30T01:59:35.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:35.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a237fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:35.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a237fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:35.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a23fff700 10 monclient(hunting): get_auth_request con 0x7f9a24098540 auth_method 0 2023-09-30T01:59:35.716 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a23fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:35.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a23fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:35.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.701+0000 7f9a23fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:35.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.702+0000 7f9a237fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:35.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.702+0000 7f9a237fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:35.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.702+0000 7f9a237fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:35.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.702+0000 7f9a23fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:35.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.702+0000 7f9a23fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:35.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.702+0000 7f9a23fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:35.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.702+0000 7f9a28d9e700 1 --2- 172.21.15.7:0/657461129 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a24137940 0x7f9a24133bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:35.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.702+0000 7f9a237fe700 10 monclient(hunting): handle_auth_done global_id 4710 payload 824 2023-09-30T01:59:35.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.702+0000 7f9a237fe700 1 -- 172.21.15.7:0/657461129 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a24098540 msgr2=0x7f9a24133150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:35.717 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.702+0000 7f9a237fe700 1 --2- 172.21.15.7:0/657461129 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a24098540 0x7f9a24133150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.702+0000 7f9a237fe700 1 -- 172.21.15.7:0/657461129 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a24137940 msgr2=0x7f9a24133bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.702+0000 7f9a237fe700 1 --2- 172.21.15.7:0/657461129 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a24137940 0x7f9a24133bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.702+0000 7f9a237fe700 10 monclient: _finish_hunting 0 2023-09-30T01:59:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.702+0000 7f9a237fe700 1 monclient: found mon.noname-a 2023-09-30T01:59:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.702+0000 7f9a237fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.702+0000 7f9a237fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.702+0000 7f9a237fe700 1 -- 172.21.15.7:0/657461129 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a14009420 con 0x7f9a241329e0 2023-09-30T01:59:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.702+0000 7f9a28d9e700 10 monclient: get_auth_request con 0x7f9a24137940 auth_method 0 2023-09-30T01:59:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.702+0000 7f9a28d9e700 1 --2- 172.21.15.7:0/657461129 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a24137940 0x7f9a24133bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:59:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a237fe700 1 -- 172.21.15.7:0/657461129 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a241329e0 msgr2=0x7f9a24133690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a237fe700 1 --2- 172.21.15.7:0/657461129 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a241329e0 0x7f9a24133690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9a2406b6b0 tx=0x7f9a14016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:35.718 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a0bfff700 1 -- 172.21.15.7:0/657461129 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1400bed0 con 0x7f9a241329e0 2023-09-30T01:59:35.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a0bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:35.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a0bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:35.719 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a0bfff700 10 monclient: dump: 2023-09-30T01:59:35.719 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:35.719 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:35.719 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:35.719 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:35.719 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:35.719 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:35.719 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:35.719 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:35.719 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:35.719 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:35.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a0bfff700 10 monclient: _finish_auth 0 2023-09-30T01:59:35.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a0bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:35.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a0bfff700 1 -- 172.21.15.7:0/657461129 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a1401c670 con 0x7f9a241329e0 2023-09-30T01:59:35.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a2959f700 5 monclient: authenticate success, global_id 4710 2023-09-30T01:59:35.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a2959f700 10 monclient: _renew_subs 2023-09-30T01:59:35.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a2959f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:35.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a2959f700 1 -- 172.21.15.7:0/657461129 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a240a35a0 con 0x7f9a241329e0 2023-09-30T01:59:35.720 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a0bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:35.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a0bfff700 1 -- 172.21.15.7:0/657461129 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1401c9f0 con 0x7f9a241329e0 2023-09-30T01:59:35.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a0bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:35.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a0bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:35.721 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a0bfff700 10 monclient: dump: 2023-09-30T01:59:35.721 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:35.721 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:35.721 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:35.721 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:35.721 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:35.721 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:35.722 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:35.722 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:35.722 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:35.722 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:35.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a2959f700 10 monclient: _renew_subs 2023-09-30T01:59:35.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a2959f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:35.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a2959f700 1 -- 172.21.15.7:0/657461129 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a241345e0 con 0x7f9a241329e0 2023-09-30T01:59:35.722 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.704+0000 7f9a227fc700 4 set_mon_vals no callback set 2023-09-30T01:59:35.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.705+0000 7f9a0bfff700 1 -- 172.21.15.7:0/657461129 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f9a1401a450 con 0x7f9a241329e0 2023-09-30T01:59:35.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.705+0000 7f9a2959f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:35.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.705+0000 7f9a2959f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:35.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.705+0000 7f9a2959f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:35.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.705+0000 7f9a2959f700 1 -- 172.21.15.7:0/657461129 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a24136630 con 0x7f9a241329e0 2023-09-30T01:59:35.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.705+0000 7f9a0bfff700 1 --2- 172.21.15.7:0/657461129 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9a100357e0 0x7f9a10037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:35.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.705+0000 7f9a0bfff700 1 -- 172.21.15.7:0/657461129 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f9a1404a340 con 0x7f9a241329e0 2023-09-30T01:59:35.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.706+0000 7f9a23fff700 1 --2- 172.21.15.7:0/657461129 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9a100357e0 0x7f9a10037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:35.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.706+0000 7f9a23fff700 10 monclient: get_auth_request con 0x7f9a100357e0 auth_method 0 2023-09-30T01:59:35.723 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.708+0000 7f9a23fff700 1 --2- 172.21.15.7:0/657461129 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9a100357e0 0x7f9a10037c90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9a180039a0 tx=0x7f9a18007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:35.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.710+0000 7f9a0bfff700 1 -- 172.21.15.7:0/657461129 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9a24136630 con 0x7f9a241329e0 2023-09-30T01:59:35.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.710+0000 7f9a0bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:35.724 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.710+0000 7f9a0bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:35.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.946+0000 7f9a2959f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:59:35.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.946+0000 7f9a2959f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:35.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.946+0000 7f9a2959f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:35.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.946+0000 7f9a2959f700 1 -- 172.21.15.7:0/657461129 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9a24136630 con 0x7f9a241329e0 2023-09-30T01:59:35.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.947+0000 7f9a0bfff700 1 -- 172.21.15.7:0/657461129 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9a24136630 con 0x7f9a241329e0 2023-09-30T01:59:35.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.947+0000 7f9a0bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:35.947 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.947+0000 7f9a0bfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:59:35.948 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:59:35.948 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:35.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.949+0000 7f9a2959f700 1 -- 172.21.15.7:0/657461129 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9a100357e0 msgr2=0x7f9a10037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:35.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.949+0000 7f9a2959f700 1 --2- 172.21.15.7:0/657461129 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9a100357e0 0x7f9a10037c90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9a180039a0 tx=0x7f9a18007ea0 comp rx=0 tx=0).stop 2023-09-30T01:59:35.949 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.949+0000 7f9a2959f700 10 monclient: shutdown 2023-09-30T01:59:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.949+0000 7f9a2959f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.949+0000 7f9a2959f700 1 -- 172.21.15.7:0/657461129 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a241329e0 msgr2=0x7f9a24133690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.949+0000 7f9a2959f700 1 --2- 172.21.15.7:0/657461129 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a241329e0 0x7f9a24133690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9a2406b6b0 tx=0x7f9a14016000 comp rx=0 tx=0).stop 2023-09-30T01:59:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.949+0000 7f9a2959f700 1 -- 172.21.15.7:0/657461129 shutdown_connections 2023-09-30T01:59:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.949+0000 7f9a2959f700 1 --2- 172.21.15.7:0/657461129 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9a100357e0 0x7f9a10037c90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.949+0000 7f9a2959f700 1 --2- 172.21.15.7:0/657461129 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a24137940 0x7f9a24133bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.949+0000 7f9a2959f700 1 --2- 172.21.15.7:0/657461129 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a24098540 0x7f9a24133150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.949+0000 7f9a2959f700 1 --2- 172.21.15.7:0/657461129 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a241329e0 0x7f9a24133690 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.949+0000 7f9a2959f700 1 -- 172.21.15.7:0/657461129 >> 172.21.15.7:0/657461129 conn(0x7f9a24071830 msgr2=0x7f9a240a1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.949+0000 7f9a2959f700 1 -- 172.21.15.7:0/657461129 shutdown_connections 2023-09-30T01:59:35.950 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:35.949+0000 7f9a2959f700 1 -- 172.21.15.7:0/657461129 wait complete. 2023-09-30T01:59:38.961 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:39.091 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.091+0000 7f9c3a327700 10 monclient: get_monmap_and_config 2023-09-30T01:59:39.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.091+0000 7f9c3a327700 10 monclient: build_initial_monmap 2023-09-30T01:59:39.092 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.091+0000 7f9c3a327700 10 monclient: monmap: 2023-09-30T01:59:39.092 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:39.092 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:39.092 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:39.092768+0000 2023-09-30T01:59:39.092 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:39.092768+0000 2023-09-30T01:59:39.092 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:39.092 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:39.092 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:39.092 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:39.092 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:39.092 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:39.093 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:39.093 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:39.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.092+0000 7f9c3a327700 1 Processor -- start 2023-09-30T01:59:39.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.092+0000 7f9c3a327700 1 -- start start 2023-09-30T01:59:39.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.092+0000 7f9c3a327700 10 monclient: init 2023-09-30T01:59:39.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.092+0000 7f9c3a327700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:39.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.092+0000 7f9c3a327700 10 monclient: _add_conns ranks=[2,0,4,1,3] 2023-09-30T01:59:39.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.092+0000 7f9c3a327700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9c34096a50 0x7f9c341327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:39.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.092+0000 7f9c3a327700 10 monclient(hunting): picked mon.noname-b con 0x7f9c34096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:39.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.092+0000 7f9c3a327700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9c340a2390 addr v1:172.21.15.7:6789/0 2023-09-30T01:59:39.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.092+0000 7f9c3a327700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f9c34097530 0x7f9c34139720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:39.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.092+0000 7f9c3a327700 10 monclient(hunting): picked mon.noname-c con 0x7f9c34097530 addr v2:172.21.15.50:3300/0 2023-09-30T01:59:39.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.092+0000 7f9c3a327700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c340980e0 con 0x7f9c340a2390 2023-09-30T01:59:39.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.092+0000 7f9c3a327700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:39.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.092+0000 7f9c3a327700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c340991a0 con 0x7f9c34097530 2023-09-30T01:59:39.097 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.092+0000 7f9c3a327700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:39.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.092+0000 7f9c3a327700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c34099480 con 0x7f9c34096a50 2023-09-30T01:59:39.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c3a327700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:39.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c3a327700 10 monclient(hunting): authenticate will time out at 1468.158011s 2023-09-30T01:59:39.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c33fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9c34096a50 0x7f9c341327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:39.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c337fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9c340a2390 0x7f9c34096e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55714/0 (socket says 172.21.15.7:55714) 2023-09-30T01:59:39.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c337fe700 1 -- 172.21.15.7:0/4060096188 learned_addr learned my addr 172.21.15.7:0/4060096188 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:59:39.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c33fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9c34096a50 0x7f9c341327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:34504/0 (socket says 172.21.15.7:34504) 2023-09-30T01:59:39.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c33fff700 10 monclient(hunting): get_auth_request con 0x7f9c34096a50 auth_method 0 2023-09-30T01:59:39.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c33fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:39.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c33fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:39.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c33fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:39.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c33fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:39.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c33fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:39.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c33fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:39.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c388c4700 1 --2- 172.21.15.7:0/4060096188 >> v2:172.21.15.50:3300/0 conn(0x7f9c34097530 0x7f9c34139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:39.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c33fff700 10 monclient(hunting): handle_auth_done global_id 4724 payload 274 2023-09-30T01:59:39.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c33fff700 1 -- 172.21.15.7:0/4060096188 >> v2:172.21.15.50:3300/0 conn(0x7f9c34097530 msgr2=0x7f9c34139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:39.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c33fff700 1 --2- 172.21.15.7:0/4060096188 >> v2:172.21.15.50:3300/0 conn(0x7f9c34097530 0x7f9c34139720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:39.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c33fff700 1 -- 172.21.15.7:0/4060096188 >> v1:172.21.15.7:6789/0 conn(0x7f9c340a2390 legacy=0x7f9c34096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:39.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c33fff700 10 monclient: _finish_hunting 0 2023-09-30T01:59:39.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c33fff700 1 monclient: found mon.noname-b 2023-09-30T01:59:39.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c33fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:39.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c33fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:39.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.093+0000 7f9c33fff700 1 -- 172.21.15.7:0/4060096188 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c34097bf0 con 0x7f9c34096a50 2023-09-30T01:59:39.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.095+0000 7f9c33fff700 1 -- 172.21.15.7:0/4060096188 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9c34096a50 msgr2=0x7f9c341327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:39.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.095+0000 7f9c33fff700 1 --2- 172.21.15.7:0/4060096188 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9c34096a50 0x7f9c341327a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9c240098e0 tx=0x7f9c240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32c4a263cbf0d448 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:39.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.095+0000 7f9c32ffd700 1 -- 172.21.15.7:0/4060096188 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2400ba30 con 0x7f9c34096a50 2023-09-30T01:59:39.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.095+0000 7f9c32ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:39.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.095+0000 7f9c32ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:39.100 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.095+0000 7f9c32ffd700 10 monclient: dump: 2023-09-30T01:59:39.100 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:39.100 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:39.100 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:39.100 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:39.100 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:39.100 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:39.101 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:39.101 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:39.101 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:39.101 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:39.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.095+0000 7f9c32ffd700 10 monclient: _finish_auth 0 2023-09-30T01:59:39.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.095+0000 7f9c32ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:39.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.095+0000 7f9c32ffd700 1 -- 172.21.15.7:0/4060096188 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2400bb90 con 0x7f9c34096a50 2023-09-30T01:59:39.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.095+0000 7f9c3a327700 5 monclient: authenticate success, global_id 4724 2023-09-30T01:59:39.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.095+0000 7f9c3a327700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:39.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.095+0000 7f9c32ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:39.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.095+0000 7f9c3a327700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:39.101 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.095+0000 7f9c32ffd700 1 -- 172.21.15.7:0/4060096188 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c24011420 con 0x7f9c34096a50 2023-09-30T01:59:39.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.095+0000 7f9c32ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:39.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.095+0000 7f9c3a327700 4 set_mon_vals no callback set 2023-09-30T01:59:39.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.095+0000 7f9c32ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:39.102 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.095+0000 7f9c32ffd700 10 monclient: dump: 2023-09-30T01:59:39.102 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:39.102 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:39.102 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:39.102 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:39.102 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:39.102 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:39.102 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:39.102 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:39.103 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:39.103 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:39.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.096+0000 7f9c3a327700 10 monclient: shutdown 2023-09-30T01:59:39.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.096+0000 7f9c3a327700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:39.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.096+0000 7f9c3a327700 1 -- 172.21.15.7:0/4060096188 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9c34096a50 msgr2=0x7f9c341327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:39.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.096+0000 7f9c3a327700 1 --2- 172.21.15.7:0/4060096188 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9c34096a50 0x7f9c341327a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9c240098e0 tx=0x7f9c240091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:39.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.096+0000 7f9c3a327700 1 -- 172.21.15.7:0/4060096188 shutdown_connections 2023-09-30T01:59:39.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.096+0000 7f9c3a327700 1 --2- 172.21.15.7:0/4060096188 >> v2:172.21.15.50:3300/0 conn(0x7f9c34097530 0x7f9c34139720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:39.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.096+0000 7f9c3a327700 1 --2- 172.21.15.7:0/4060096188 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9c34096a50 0x7f9c341327a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:39.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.096+0000 7f9c3a327700 1 -- 172.21.15.7:0/4060096188 >> 172.21.15.7:0/4060096188 conn(0x7f9c34070cf0 msgr2=0x7f9c340710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:39.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.096+0000 7f9c3a327700 1 -- 172.21.15.7:0/4060096188 shutdown_connections 2023-09-30T01:59:39.103 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.096+0000 7f9c3a327700 1 -- 172.21.15.7:0/4060096188 wait complete. 2023-09-30T01:59:39.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.097+0000 7f9c3a327700 10 monclient: build_initial_monmap 2023-09-30T01:59:39.104 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.097+0000 7f9c3a327700 10 monclient: monmap: 2023-09-30T01:59:39.104 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:39.104 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:39.104 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:39.104 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:39.104 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:39.104 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:39.104 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:39.104 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:39.104 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:39.104 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:39.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.097+0000 7f9c3a327700 1 Processor -- start 2023-09-30T01:59:39.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.097+0000 7f9c3a327700 1 -- start start 2023-09-30T01:59:39.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.097+0000 7f9c3a327700 10 monclient: init 2023-09-30T01:59:39.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c3a327700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:39.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c3a327700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:59:39.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c3a327700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c34096a50 0x7f9c3409b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:39.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c3a327700 10 monclient(hunting): picked mon.noname-b con 0x7f9c34096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:39.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c3a327700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9c34097530 0x7f9c3409bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:39.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c3a327700 10 monclient(hunting): picked mon.noname-c con 0x7f9c34097530 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:39.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c3a327700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c340a2390 0x7f9c3409c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c3a327700 10 monclient(hunting): picked mon.noname-a con 0x7f9c340a2390 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c3a327700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c3a327700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3409df70 con 0x7f9c340a2390 2023-09-30T01:59:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c3a327700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c3a327700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3409e890 con 0x7f9c34096a50 2023-09-30T01:59:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c3a327700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c3a327700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c341334e0 con 0x7f9c34097530 2023-09-30T01:59:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c3a327700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c3a327700 10 monclient(hunting): authenticate will time out at 1468.163093s 2023-09-30T01:59:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c337fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9c34097530 0x7f9c3409bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c388c4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c340a2390 0x7f9c3409c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:39.110 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c337fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9c34097530 0x7f9c3409bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:34514/0 (socket says 172.21.15.7:34514) 2023-09-30T01:59:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c337fe700 1 -- 172.21.15.7:0/2693255690 learned_addr learned my addr 172.21.15.7:0/2693255690 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c337fe700 10 monclient(hunting): get_auth_request con 0x7f9c34097530 auth_method 0 2023-09-30T01:59:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c337fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c337fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c388c4700 10 monclient(hunting): get_auth_request con 0x7f9c340a2390 auth_method 0 2023-09-30T01:59:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c388c4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c388c4700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c388c4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:39.111 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:39.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c388c4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:39.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c388c4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:39.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c388c4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:39.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c33fff700 1 --2- 172.21.15.7:0/2693255690 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c34096a50 0x7f9c3409b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:39.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c33fff700 10 monclient(hunting): get_auth_request con 0x7f9c34096a50 auth_method 0 2023-09-30T01:59:39.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c33fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:39.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c33fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:39.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c33fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:39.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c337fe700 10 monclient(hunting): handle_auth_done global_id 4727 payload 824 2023-09-30T01:59:39.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c337fe700 1 -- 172.21.15.7:0/2693255690 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c34096a50 msgr2=0x7f9c3409b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:39.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c337fe700 1 --2- 172.21.15.7:0/2693255690 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c34096a50 0x7f9c3409b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:39.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c337fe700 1 -- 172.21.15.7:0/2693255690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c340a2390 msgr2=0x7f9c3409c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:39.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c337fe700 1 --2- 172.21.15.7:0/2693255690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c340a2390 0x7f9c3409c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:39.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c337fe700 10 monclient: _finish_hunting 0 2023-09-30T01:59:39.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c337fe700 1 monclient: found mon.noname-c 2023-09-30T01:59:39.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c337fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:39.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c337fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:59:39.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c337fe700 1 -- 172.21.15.7:0/2693255690 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c24009420 con 0x7f9c34097530 2023-09-30T01:59:39.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.098+0000 7f9c388c4700 1 --2- 172.21.15.7:0/2693255690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c340a2390 0x7f9c3409c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:39.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.100+0000 7f9c337fe700 1 -- 172.21.15.7:0/2693255690 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9c34097530 msgr2=0x7f9c3409bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:39.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.100+0000 7f9c337fe700 1 --2- 172.21.15.7:0/2693255690 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9c34097530 0x7f9c3409bb90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9c2800c640 tx=0x7f9c28004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:39.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.100+0000 7f9c1bfff700 1 -- 172.21.15.7:0/2693255690 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c28005d40 con 0x7f9c34097530 2023-09-30T01:59:39.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.100+0000 7f9c1bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:39.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.100+0000 7f9c1bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:59:39.114 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.100+0000 7f9c1bfff700 10 monclient: dump: 2023-09-30T01:59:39.114 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:39.114 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:39.114 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:39.114 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:39.114 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:39.114 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:39.114 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:39.114 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:39.114 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:39.114 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:39.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.100+0000 7f9c1bfff700 10 monclient: _finish_auth 0 2023-09-30T01:59:39.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.100+0000 7f9c1bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:39.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.100+0000 7f9c3a327700 5 monclient: authenticate success, global_id 4727 2023-09-30T01:59:39.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.100+0000 7f9c1bfff700 1 -- 172.21.15.7:0/2693255690 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c280070d0 con 0x7f9c34097530 2023-09-30T01:59:39.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.100+0000 7f9c1bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:39.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.100+0000 7f9c3a327700 10 monclient: _renew_subs 2023-09-30T01:59:39.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.100+0000 7f9c1bfff700 1 -- 172.21.15.7:0/2693255690 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c28007490 con 0x7f9c34097530 2023-09-30T01:59:39.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.100+0000 7f9c3a327700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:39.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.100+0000 7f9c3a327700 1 -- 172.21.15.7:0/2693255690 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c34130120 con 0x7f9c34097530 2023-09-30T01:59:39.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.100+0000 7f9c1bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:39.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.100+0000 7f9c1bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:39.115 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.100+0000 7f9c327fc700 4 set_mon_vals no callback set 2023-09-30T01:59:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.100+0000 7f9c1bfff700 10 monclient: dump: 2023-09-30T01:59:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:39.116 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.101+0000 7f9c3a327700 10 monclient: _renew_subs 2023-09-30T01:59:39.116 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.101+0000 7f9c3a327700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.101+0000 7f9c3a327700 1 -- 172.21.15.7:0/2693255690 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c3409f040 con 0x7f9c34097530 2023-09-30T01:59:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.102+0000 7f9c1bfff700 1 -- 172.21.15.7:0/2693255690 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f9c2801b3b0 con 0x7f9c34097530 2023-09-30T01:59:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.102+0000 7f9c3a327700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.102+0000 7f9c3a327700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.102+0000 7f9c3a327700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.102+0000 7f9c3a327700 1 -- 172.21.15.7:0/2693255690 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c34064c10 con 0x7f9c34097530 2023-09-30T01:59:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.102+0000 7f9c1bfff700 1 --2- 172.21.15.7:0/2693255690 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9c200357e0 0x7f9c20037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.102+0000 7f9c1bfff700 1 -- 172.21.15.7:0/2693255690 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f9c28047220 con 0x7f9c34097530 2023-09-30T01:59:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.102+0000 7f9c33fff700 1 --2- 172.21.15.7:0/2693255690 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9c200357e0 0x7f9c20037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.102+0000 7f9c33fff700 10 monclient: get_auth_request con 0x7f9c200357e0 auth_method 0 2023-09-30T01:59:39.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.105+0000 7f9c33fff700 1 --2- 172.21.15.7:0/2693255690 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9c200357e0 0x7f9c20037c90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9c240030c0 tx=0x7f9c24016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.106+0000 7f9c1bfff700 1 -- 172.21.15.7:0/2693255690 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9c28018260 con 0x7f9c34097530 2023-09-30T01:59:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.106+0000 7f9c1bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:39.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.106+0000 7f9c1bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:39.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.343+0000 7f9c3a327700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:59:39.343 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.343+0000 7f9c3a327700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:39.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.343+0000 7f9c3a327700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:39.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.343+0000 7f9c3a327700 1 -- 172.21.15.7:0/2693255690 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9c34132d60 con 0x7f9c34097530 2023-09-30T01:59:39.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.343+0000 7f9c1bfff700 1 -- 172.21.15.7:0/2693255690 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9c28017100 con 0x7f9c34097530 2023-09-30T01:59:39.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.343+0000 7f9c1bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:39.344 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.343+0000 7f9c1bfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:59:39.344 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:59:39.344 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:39.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.345+0000 7f9c3a327700 1 -- 172.21.15.7:0/2693255690 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9c200357e0 msgr2=0x7f9c20037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:39.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.345+0000 7f9c3a327700 1 --2- 172.21.15.7:0/2693255690 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9c200357e0 0x7f9c20037c90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9c240030c0 tx=0x7f9c24016000 comp rx=0 tx=0).stop 2023-09-30T01:59:39.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.345+0000 7f9c3a327700 10 monclient: shutdown 2023-09-30T01:59:39.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.345+0000 7f9c3a327700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:39.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.345+0000 7f9c3a327700 1 -- 172.21.15.7:0/2693255690 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9c34097530 msgr2=0x7f9c3409bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:39.346 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.345+0000 7f9c3a327700 1 --2- 172.21.15.7:0/2693255690 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9c34097530 0x7f9c3409bb90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9c2800c640 tx=0x7f9c28004a80 comp rx=0 tx=0).stop 2023-09-30T01:59:39.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.345+0000 7f9c3a327700 1 -- 172.21.15.7:0/2693255690 shutdown_connections 2023-09-30T01:59:39.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.345+0000 7f9c3a327700 1 --2- 172.21.15.7:0/2693255690 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9c200357e0 0x7f9c20037c90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:39.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.345+0000 7f9c3a327700 1 --2- 172.21.15.7:0/2693255690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c340a2390 0x7f9c3409c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:39.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.345+0000 7f9c3a327700 1 --2- 172.21.15.7:0/2693255690 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9c34096a50 0x7f9c3409b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:39.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.345+0000 7f9c3a327700 1 --2- 172.21.15.7:0/2693255690 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9c34097530 0x7f9c3409bb90 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:39.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.345+0000 7f9c3a327700 1 -- 172.21.15.7:0/2693255690 >> 172.21.15.7:0/2693255690 conn(0x7f9c34070cf0 msgr2=0x7f9c340a4500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:39.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.345+0000 7f9c3a327700 1 -- 172.21.15.7:0/2693255690 shutdown_connections 2023-09-30T01:59:39.347 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:39.345+0000 7f9c3a327700 1 -- 172.21.15.7:0/2693255690 wait complete. 2023-09-30T01:59:42.358 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:42.489 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.489+0000 7f24b3112700 10 monclient: get_monmap_and_config 2023-09-30T01:59:42.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.489+0000 7f24b3112700 10 monclient: build_initial_monmap 2023-09-30T01:59:42.490 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.489+0000 7f24b3112700 10 monclient: monmap: 2023-09-30T01:59:42.490 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:42.490 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:42.490 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:42.490971+0000 2023-09-30T01:59:42.490 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:42.490971+0000 2023-09-30T01:59:42.490 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:42.490 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:42.490 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:42.490 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:42.491 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:42.491 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:42.491 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:42.491 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:42.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.490+0000 7f24b3112700 1 Processor -- start 2023-09-30T01:59:42.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.490+0000 7f24b3112700 1 -- start start 2023-09-30T01:59:42.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.490+0000 7f24b3112700 10 monclient: init 2023-09-30T01:59:42.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24b3112700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:42.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24b3112700 10 monclient: _add_conns ranks=[0,2,1,3,4] 2023-09-30T01:59:42.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24b3112700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f24ac096a50 addr v1:172.21.15.7:6789/0 2023-09-30T01:59:42.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24b3112700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f24ac0a2390 0x7f24ac096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:42.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24b3112700 10 monclient(hunting): picked mon.noname-b con 0x7f24ac0a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:42.493 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24b3112700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f24ac097360 addr v1:172.21.15.50:6789/0 2023-09-30T01:59:42.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24b3112700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ac0980e0 con 0x7f24ac096a50 2023-09-30T01:59:42.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24b3112700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ac132eb0 con 0x7f24ac097360 2023-09-30T01:59:42.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24b3112700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:42.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24b3112700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24ac0991a0 con 0x7f24ac0a2390 2023-09-30T01:59:42.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24b3112700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:42.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24b3112700 10 monclient(hunting): authenticate will time out at 1471.556230s 2023-09-30T01:59:42.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24abfff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f24ac0a2390 0x7f24ac096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:42.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24b0eae700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f24ac096a50 0x7f24ac1327a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55726/0 (socket says 172.21.15.7:55726) 2023-09-30T01:59:42.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24b0eae700 1 -- 172.21.15.7:0/4135622233 learned_addr learned my addr 172.21.15.7:0/4135622233 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:59:42.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24abfff700 10 monclient(hunting): get_auth_request con 0x7f24ac0a2390 auth_method 0 2023-09-30T01:59:42.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:42.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24abfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:42.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24abfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:42.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24abfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:42.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24abfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:42.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24abfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:42.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24abfff700 10 monclient(hunting): handle_auth_done global_id 4730 payload 274 2023-09-30T01:59:42.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24abfff700 1 -- 172.21.15.7:0/4135622233 >> v1:172.21.15.50:6789/0 conn(0x7f24ac097360 legacy=0x7f24ac139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:42.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24abfff700 1 -- 172.21.15.7:0/4135622233 >> v1:172.21.15.7:6789/0 conn(0x7f24ac096a50 legacy=0x7f24ac1327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:42.495 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24abfff700 10 monclient: _finish_hunting 0 2023-09-30T01:59:42.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24abfff700 1 monclient: found mon.noname-b 2023-09-30T01:59:42.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24abfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:42.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24abfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:42.496 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.491+0000 7f24abfff700 1 -- 172.21.15.7:0/4135622233 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ac099480 con 0x7f24ac0a2390 2023-09-30T01:59:42.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.493+0000 7f24abfff700 1 -- 172.21.15.7:0/4135622233 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f24ac0a2390 msgr2=0x7f24ac096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:42.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.493+0000 7f24abfff700 1 --2- 172.21.15.7:0/4135622233 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f24ac0a2390 0x7f24ac096e20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f24a0009910 tx=0x7f24a00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b483cd76203f8c76 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:42.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24ab7fe700 1 -- 172.21.15.7:0/4135622233 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a000b9f0 con 0x7f24ac0a2390 2023-09-30T01:59:42.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24ab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:42.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24ab7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:42.502 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24ab7fe700 10 monclient: dump: 2023-09-30T01:59:42.502 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:42.502 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:42.502 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:42.503 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:42.503 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:42.503 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:42.503 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:42.503 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:42.503 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:42.503 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:42.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24ab7fe700 10 monclient: _finish_auth 0 2023-09-30T01:59:42.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24ab7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:42.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24b3112700 5 monclient: authenticate success, global_id 4730 2023-09-30T01:59:42.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24ab7fe700 1 -- 172.21.15.7:0/4135622233 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a000bb50 con 0x7f24ac0a2390 2023-09-30T01:59:42.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24b3112700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:42.503 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24ab7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:42.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24b3112700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:42.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24b3112700 4 set_mon_vals no callback set 2023-09-30T01:59:42.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24ab7fe700 1 -- 172.21.15.7:0/4135622233 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a0011420 con 0x7f24ac0a2390 2023-09-30T01:59:42.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24ab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:42.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24ab7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:42.504 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24ab7fe700 10 monclient: dump: 2023-09-30T01:59:42.504 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:42.504 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:42.504 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:42.504 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:42.504 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:42.504 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:42.505 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:42.505 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:42.505 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:42.505 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:42.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24b3112700 10 monclient: shutdown 2023-09-30T01:59:42.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24b3112700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:42.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24b3112700 1 -- 172.21.15.7:0/4135622233 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f24ac0a2390 msgr2=0x7f24ac096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:42.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24b3112700 1 --2- 172.21.15.7:0/4135622233 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f24ac0a2390 0x7f24ac096e20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f24a0009910 tx=0x7f24a00091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:42.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24b3112700 1 -- 172.21.15.7:0/4135622233 shutdown_connections 2023-09-30T01:59:42.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24b3112700 1 --2- 172.21.15.7:0/4135622233 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f24ac0a2390 0x7f24ac096e20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:42.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24b3112700 1 -- 172.21.15.7:0/4135622233 >> 172.21.15.7:0/4135622233 conn(0x7f24ac070cf0 msgr2=0x7f24ac0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:42.505 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24b3112700 1 -- 172.21.15.7:0/4135622233 shutdown_connections 2023-09-30T01:59:42.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.494+0000 7f24b3112700 1 -- 172.21.15.7:0/4135622233 wait complete. 2023-09-30T01:59:42.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.495+0000 7f24b3112700 10 monclient: build_initial_monmap 2023-09-30T01:59:42.506 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.495+0000 7f24b3112700 10 monclient: monmap: 2023-09-30T01:59:42.506 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:42.506 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:42.506 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:42.506 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:42.506 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:42.506 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:42.506 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:42.506 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:42.506 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:42.507 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:42.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.495+0000 7f24b3112700 1 Processor -- start 2023-09-30T01:59:42.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.495+0000 7f24b3112700 1 -- start start 2023-09-30T01:59:42.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.495+0000 7f24b3112700 10 monclient: init 2023-09-30T01:59:42.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b3112700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:42.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b3112700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:59:42.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b3112700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24ac096a50 0x7f24ac09baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:42.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b3112700 10 monclient(hunting): picked mon.noname-a con 0x7f24ac096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:42.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b3112700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24ac097360 0x7f24ac09bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:42.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b3112700 10 monclient(hunting): picked mon.noname-b con 0x7f24ac097360 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:42.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b3112700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f24ac0a2390 0x7f24ac09c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:42.507 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b3112700 10 monclient(hunting): picked mon.noname-c con 0x7f24ac0a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:42.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b3112700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:42.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b3112700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ac132400 con 0x7f24ac096a50 2023-09-30T01:59:42.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b3112700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:42.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b3112700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ac09f1d0 con 0x7f24ac097360 2023-09-30T01:59:42.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b3112700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:42.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b3112700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24ac09ec30 con 0x7f24ac0a2390 2023-09-30T01:59:42.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b3112700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:42.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b3112700 10 monclient(hunting): authenticate will time out at 1471.561352s 2023-09-30T01:59:42.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b0eae700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24ac096a50 0x7f24ac09baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:42.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b0eae700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24ac096a50 0x7f24ac09baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46778/0 (socket says 172.21.15.7:46778) 2023-09-30T01:59:42.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b0eae700 1 -- 172.21.15.7:0/3288332702 learned_addr learned my addr 172.21.15.7:0/3288332702 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:42.508 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b0eae700 10 monclient(hunting): get_auth_request con 0x7f24ac096a50 auth_method 0 2023-09-30T01:59:42.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b0eae700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:42.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b0eae700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:42.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b0eae700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:42.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b16af700 1 --2- 172.21.15.7:0/3288332702 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f24ac0a2390 0x7f24ac09c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:42.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b16af700 10 monclient(hunting): get_auth_request con 0x7f24ac0a2390 auth_method 0 2023-09-30T01:59:42.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b16af700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:42.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b16af700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:42.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b16af700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:42.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b0eae700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:42.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b0eae700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:42.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b0eae700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:42.509 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b16af700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:42.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b16af700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:42.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b16af700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:42.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24abfff700 1 --2- 172.21.15.7:0/3288332702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24ac097360 0x7f24ac09bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:42.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b0eae700 10 monclient(hunting): handle_auth_done global_id 4716 payload 824 2023-09-30T01:59:42.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b0eae700 1 -- 172.21.15.7:0/3288332702 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f24ac0a2390 msgr2=0x7f24ac09c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:42.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b0eae700 1 --2- 172.21.15.7:0/3288332702 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f24ac0a2390 0x7f24ac09c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:42.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b0eae700 1 -- 172.21.15.7:0/3288332702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24ac097360 msgr2=0x7f24ac09bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:42.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b0eae700 1 --2- 172.21.15.7:0/3288332702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24ac097360 0x7f24ac09bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:42.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b0eae700 10 monclient: _finish_hunting 0 2023-09-30T01:59:42.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b0eae700 1 monclient: found mon.noname-a 2023-09-30T01:59:42.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b0eae700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:42.510 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b0eae700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:42.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b0eae700 1 -- 172.21.15.7:0/3288332702 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a0009420 con 0x7f24ac096a50 2023-09-30T01:59:42.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24b16af700 1 --2- 172.21.15.7:0/3288332702 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f24ac0a2390 0x7f24ac09c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:42.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24abfff700 10 monclient: get_auth_request con 0x7f24ac097360 auth_method 0 2023-09-30T01:59:42.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.496+0000 7f24abfff700 1 --2- 172.21.15.7:0/3288332702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24ac097360 0x7f24ac09bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:59:42.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24b0eae700 1 -- 172.21.15.7:0/3288332702 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24ac096a50 msgr2=0x7f24ac09baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:42.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24b0eae700 1 --2- 172.21.15.7:0/3288332702 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24ac096a50 0x7f24ac09baa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f249c00c5e0 tx=0x7f249c004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:42.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24a8ff9700 1 -- 172.21.15.7:0/3288332702 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c005d40 con 0x7f24ac096a50 2023-09-30T01:59:42.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24a8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:42.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24a8ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:42.511 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24a8ff9700 10 monclient: dump: 2023-09-30T01:59:42.511 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:42.511 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:42.512 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:42.512 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:42.512 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:42.512 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:42.512 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:42.512 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:42.512 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:42.512 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:42.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24a8ff9700 10 monclient: _finish_auth 0 2023-09-30T01:59:42.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24a8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:42.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24a8ff9700 1 -- 172.21.15.7:0/3288332702 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f249c0070d0 con 0x7f24ac096a50 2023-09-30T01:59:42.512 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24b3112700 5 monclient: authenticate success, global_id 4716 2023-09-30T01:59:42.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24b3112700 10 monclient: _renew_subs 2023-09-30T01:59:42.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24b3112700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:42.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24b3112700 1 -- 172.21.15.7:0/3288332702 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24ac09ef50 con 0x7f24ac096a50 2023-09-30T01:59:42.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24a8ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:42.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24a8ff9700 1 -- 172.21.15.7:0/3288332702 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c007490 con 0x7f24ac096a50 2023-09-30T01:59:42.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24a8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:42.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24a8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:42.513 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24a8ff9700 10 monclient: dump: 2023-09-30T01:59:42.513 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:42.513 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:42.513 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:42.513 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:42.514 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:42.514 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:42.514 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:42.514 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:42.514 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:42.514 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:42.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24aaffd700 4 set_mon_vals no callback set 2023-09-30T01:59:42.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24b3112700 10 monclient: _renew_subs 2023-09-30T01:59:42.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24b3112700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:42.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.499+0000 7f24b3112700 1 -- 172.21.15.7:0/3288332702 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24ac09e000 con 0x7f24ac096a50 2023-09-30T01:59:42.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.500+0000 7f24a8ff9700 1 -- 172.21.15.7:0/3288332702 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f249c01b3b0 con 0x7f24ac096a50 2023-09-30T01:59:42.514 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.500+0000 7f24b3112700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:42.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.500+0000 7f24b3112700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:42.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.500+0000 7f24b3112700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:42.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.500+0000 7f24b3112700 1 -- 172.21.15.7:0/3288332702 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24ac0a4c20 con 0x7f24ac096a50 2023-09-30T01:59:42.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.500+0000 7f24a8ff9700 1 --2- 172.21.15.7:0/3288332702 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f24980357e0 0x7f2498037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:42.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.500+0000 7f24a8ff9700 1 -- 172.21.15.7:0/3288332702 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f249c047260 con 0x7f24ac096a50 2023-09-30T01:59:42.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.500+0000 7f24abfff700 1 --2- 172.21.15.7:0/3288332702 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f24980357e0 0x7f2498037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:42.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.500+0000 7f24abfff700 10 monclient: get_auth_request con 0x7f24980357e0 auth_method 0 2023-09-30T01:59:42.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.503+0000 7f24abfff700 1 --2- 172.21.15.7:0/3288332702 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f24980357e0 0x7f2498037c90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f24ac06a090 tx=0x7f24a0016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:42.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.505+0000 7f24a8ff9700 1 -- 172.21.15.7:0/3288332702 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f249c018260 con 0x7f24ac096a50 2023-09-30T01:59:42.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.505+0000 7f24a8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:42.516 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.505+0000 7f24a8ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:42.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.741+0000 7f24b3112700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:59:42.741 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.741+0000 7f24b3112700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:42.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.741+0000 7f24b3112700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:42.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.741+0000 7f24b3112700 1 -- 172.21.15.7:0/3288332702 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f24ac064c10 con 0x7f24ac096a50 2023-09-30T01:59:42.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.741+0000 7f24a8ff9700 1 -- 172.21.15.7:0/3288332702 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f249c017100 con 0x7f24ac096a50 2023-09-30T01:59:42.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.741+0000 7f24a8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:42.742 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.741+0000 7f24a8ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:59:42.742 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:59:42.742 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:42.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.743+0000 7f24b3112700 1 -- 172.21.15.7:0/3288332702 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f24980357e0 msgr2=0x7f2498037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:42.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.744+0000 7f24b3112700 1 --2- 172.21.15.7:0/3288332702 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f24980357e0 0x7f2498037c90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f24ac06a090 tx=0x7f24a0016000 comp rx=0 tx=0).stop 2023-09-30T01:59:42.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.744+0000 7f24b3112700 10 monclient: shutdown 2023-09-30T01:59:42.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.744+0000 7f24b3112700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:42.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.744+0000 7f24b3112700 1 -- 172.21.15.7:0/3288332702 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24ac096a50 msgr2=0x7f24ac09baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:42.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.744+0000 7f24b3112700 1 --2- 172.21.15.7:0/3288332702 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24ac096a50 0x7f24ac09baa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f249c00c5e0 tx=0x7f249c004a80 comp rx=0 tx=0).stop 2023-09-30T01:59:42.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.744+0000 7f24b3112700 1 -- 172.21.15.7:0/3288332702 shutdown_connections 2023-09-30T01:59:42.744 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.744+0000 7f24b3112700 1 --2- 172.21.15.7:0/3288332702 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f24980357e0 0x7f2498037c90 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:42.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.744+0000 7f24b3112700 1 --2- 172.21.15.7:0/3288332702 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24ac096a50 0x7f24ac09baa0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:42.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.744+0000 7f24b3112700 1 --2- 172.21.15.7:0/3288332702 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f24ac0a2390 0x7f24ac09c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:42.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.744+0000 7f24b3112700 1 --2- 172.21.15.7:0/3288332702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24ac097360 0x7f24ac09bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:42.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.744+0000 7f24b3112700 1 -- 172.21.15.7:0/3288332702 >> 172.21.15.7:0/3288332702 conn(0x7f24ac070cf0 msgr2=0x7f24ac0a3670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:42.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.744+0000 7f24b3112700 1 -- 172.21.15.7:0/3288332702 shutdown_connections 2023-09-30T01:59:42.745 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:42.744+0000 7f24b3112700 1 -- 172.21.15.7:0/3288332702 wait complete. 2023-09-30T01:59:45.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:45.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.885+0000 7fccc047e700 10 monclient: get_monmap_and_config 2023-09-30T01:59:45.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.885+0000 7fccc047e700 10 monclient: build_initial_monmap 2023-09-30T01:59:45.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.885+0000 7fccc047e700 10 monclient: monmap: 2023-09-30T01:59:45.886 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:45.886 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:45.886 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:45.886708+0000 2023-09-30T01:59:45.886 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:45.886708+0000 2023-09-30T01:59:45.886 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:45.886 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:45.886 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:45.886 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:45.886 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:45.886 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:45.886 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:45.887 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:45.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.886+0000 7fccc047e700 1 Processor -- start 2023-09-30T01:59:45.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.886+0000 7fccc047e700 1 -- start start 2023-09-30T01:59:45.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.886+0000 7fccc047e700 10 monclient: init 2023-09-30T01:59:45.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.886+0000 7fccc047e700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:45.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.886+0000 7fccc047e700 10 monclient: _add_conns ranks=[4,0,2,3,1] 2023-09-30T01:59:45.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.886+0000 7fccc047e700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fccb80a29e0 0x7fccb8096a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:45.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.886+0000 7fccc047e700 10 monclient(hunting): picked mon.noname-c con 0x7fccb80a29e0 addr v2:172.21.15.50:3300/0 2023-09-30T01:59:45.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.886+0000 7fccc047e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fccb8096f90 addr v1:172.21.15.7:6789/0 2023-09-30T01:59:45.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccc047e700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fccb8097390 0x7fccb8132930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:45.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccc047e700 10 monclient(hunting): picked mon.noname-b con 0x7fccb8097390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:45.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccc047e700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb8069cc0 con 0x7fccb8096f90 2023-09-30T01:59:45.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccc047e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:45.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccc047e700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb8098ea0 con 0x7fccb80a29e0 2023-09-30T01:59:45.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccc047e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:45.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccc047e700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb80a2db0 con 0x7fccb8097390 2023-09-30T01:59:45.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccc047e700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:45.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccc047e700 10 monclient(hunting): authenticate will time out at 1474.952055s 2023-09-30T01:59:45.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbda19700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fccb8096f90 0x7fccb8132220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46778/0 (socket says 172.21.15.7:46778) 2023-09-30T01:59:45.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbda19700 1 -- 172.21.15.7:0/959699904 learned_addr learned my addr 172.21.15.7:0/959699904 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:59:45.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbea1b700 1 --2- 172.21.15.7:0/959699904 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fccb8097390 0x7fccb8132930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:45.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbea1b700 10 monclient(hunting): get_auth_request con 0x7fccb8097390 auth_method 0 2023-09-30T01:59:45.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbea1b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:45.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbea1b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:45.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbea1b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:45.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbea1b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:45.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbea1b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:45.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbea1b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:45.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbe21a700 1 --2- 172.21.15.7:0/959699904 >> v2:172.21.15.50:3300/0 conn(0x7fccb80a29e0 0x7fccb8096a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:45.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbe21a700 10 monclient(hunting): get_auth_request con 0x7fccb80a29e0 auth_method 0 2023-09-30T01:59:45.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbe21a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:45.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbe21a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:45.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbe21a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:45.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbea1b700 10 monclient(hunting): handle_auth_done global_id 4736 payload 274 2023-09-30T01:59:45.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbea1b700 1 -- 172.21.15.7:0/959699904 >> v2:172.21.15.50:3300/0 conn(0x7fccb80a29e0 msgr2=0x7fccb8096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:45.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbea1b700 1 --2- 172.21.15.7:0/959699904 >> v2:172.21.15.50:3300/0 conn(0x7fccb80a29e0 0x7fccb8096a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:45.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbea1b700 1 -- 172.21.15.7:0/959699904 >> v1:172.21.15.7:6789/0 conn(0x7fccb8096f90 legacy=0x7fccb8132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:45.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbea1b700 10 monclient: _finish_hunting 0 2023-09-30T01:59:45.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbea1b700 1 monclient: found mon.noname-b 2023-09-30T01:59:45.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbea1b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:45.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbea1b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:45.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.887+0000 7fccbea1b700 1 -- 172.21.15.7:0/959699904 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb8098270 con 0x7fccb8097390 2023-09-30T01:59:45.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.889+0000 7fccbea1b700 1 -- 172.21.15.7:0/959699904 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fccb8097390 msgr2=0x7fccb8132930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:45.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.889+0000 7fccbea1b700 1 --2- 172.21.15.7:0/959699904 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fccb8097390 0x7fccb8132930 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fccb00098e0 tx=0x7fccb00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d7bfac8772d848a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:45.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.889+0000 7fccbd218700 1 -- 172.21.15.7:0/959699904 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb0009e80 con 0x7fccb8097390 2023-09-30T01:59:45.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.889+0000 7fccbd218700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:45.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.889+0000 7fccbd218700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:45.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.889+0000 7fccbd218700 10 monclient: dump: 2023-09-30T01:59:45.895 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:45.895 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:45.895 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:45.895 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:45.895 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:45.895 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:45.895 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:45.895 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:45.896 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:45.896 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:45.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.889+0000 7fccbd218700 10 monclient: _finish_auth 0 2023-09-30T01:59:45.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.889+0000 7fccbd218700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:45.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.889+0000 7fccbd218700 1 -- 172.21.15.7:0/959699904 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb000ba30 con 0x7fccb8097390 2023-09-30T01:59:45.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.889+0000 7fccc047e700 5 monclient: authenticate success, global_id 4736 2023-09-30T01:59:45.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.889+0000 7fccc047e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:45.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.889+0000 7fccbd218700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:45.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.889+0000 7fccc047e700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:45.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.889+0000 7fccbd218700 1 -- 172.21.15.7:0/959699904 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb000bdb0 con 0x7fccb8097390 2023-09-30T01:59:45.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.889+0000 7fccc047e700 4 set_mon_vals no callback set 2023-09-30T01:59:45.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.889+0000 7fccbd218700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:45.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.889+0000 7fccbd218700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:45.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.889+0000 7fccbd218700 10 monclient: dump: 2023-09-30T01:59:45.897 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:45.897 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:45.897 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:45.897 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:45.897 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:45.897 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:45.897 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:45.897 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:45.897 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:45.897 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:45.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.890+0000 7fccc047e700 10 monclient: shutdown 2023-09-30T01:59:45.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.890+0000 7fccc047e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:45.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.890+0000 7fccc047e700 1 -- 172.21.15.7:0/959699904 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fccb8097390 msgr2=0x7fccb8132930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:45.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.890+0000 7fccc047e700 1 --2- 172.21.15.7:0/959699904 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fccb8097390 0x7fccb8132930 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fccb00098e0 tx=0x7fccb00091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:45.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.890+0000 7fccc047e700 1 -- 172.21.15.7:0/959699904 shutdown_connections 2023-09-30T01:59:45.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.890+0000 7fccc047e700 1 --2- 172.21.15.7:0/959699904 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fccb8097390 0x7fccb8132930 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:45.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.890+0000 7fccc047e700 1 --2- 172.21.15.7:0/959699904 >> v2:172.21.15.50:3300/0 conn(0x7fccb80a29e0 0x7fccb8096a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:45.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.890+0000 7fccc047e700 1 -- 172.21.15.7:0/959699904 >> 172.21.15.7:0/959699904 conn(0x7fccb80710d0 msgr2=0x7fccb80714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:45.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.890+0000 7fccc047e700 1 -- 172.21.15.7:0/959699904 shutdown_connections 2023-09-30T01:59:45.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.890+0000 7fccc047e700 1 -- 172.21.15.7:0/959699904 wait complete. 2023-09-30T01:59:45.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.891+0000 7fccc047e700 10 monclient: build_initial_monmap 2023-09-30T01:59:45.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.891+0000 7fccc047e700 10 monclient: monmap: 2023-09-30T01:59:45.899 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:45.899 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:45.899 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:45.899 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:45.899 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:45.899 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:45.899 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:45.899 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:45.899 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:45.899 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:45.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.891+0000 7fccc047e700 1 Processor -- start 2023-09-30T01:59:45.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.891+0000 7fccc047e700 1 -- start start 2023-09-30T01:59:45.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.891+0000 7fccc047e700 10 monclient: init 2023-09-30T01:59:45.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.891+0000 7fccc047e700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:45.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.891+0000 7fccc047e700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:59:45.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.891+0000 7fccc047e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccb8096f90 0x7fccb8141480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:45.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.891+0000 7fccc047e700 10 monclient(hunting): picked mon.noname-a con 0x7fccb8096f90 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:45.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.891+0000 7fccc047e700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fccb8097390 0x7fccb812d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:45.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.891+0000 7fccc047e700 10 monclient(hunting): picked mon.noname-c con 0x7fccb8097390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:45.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.891+0000 7fccc047e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fccb80a29e0 0x7fccb809ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:45.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.891+0000 7fccc047e700 10 monclient(hunting): picked mon.noname-b con 0x7fccb80a29e0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:45.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.891+0000 7fccc047e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:45.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.891+0000 7fccc047e700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb80a0950 con 0x7fccb8096f90 2023-09-30T01:59:45.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.891+0000 7fccc047e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:45.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.891+0000 7fccc047e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb809f030 con 0x7fccb80a29e0 2023-09-30T01:59:45.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.891+0000 7fccc047e700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:45.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.891+0000 7fccc047e700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb80a05e0 con 0x7fccb8097390 2023-09-30T01:59:45.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.891+0000 7fccc047e700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:45.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.891+0000 7fccc047e700 10 monclient(hunting): authenticate will time out at 1474.956967s 2023-09-30T01:59:45.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbda19700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fccb8097390 0x7fccb812d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:45.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbe21a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccb8096f90 0x7fccb8141480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:45.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbe21a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccb8096f90 0x7fccb8141480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54382/0 (socket says 172.21.15.7:54382) 2023-09-30T01:59:45.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbe21a700 1 -- 172.21.15.7:0/4044195332 learned_addr learned my addr 172.21.15.7:0/4044195332 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:45.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbda19700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fccb8097390 0x7fccb812d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:40538/0 (socket says 172.21.15.7:40538) 2023-09-30T01:59:45.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbe21a700 10 monclient(hunting): get_auth_request con 0x7fccb8096f90 auth_method 0 2023-09-30T01:59:45.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbe21a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:45.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbe21a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:45.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbe21a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:45.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbda19700 10 monclient(hunting): get_auth_request con 0x7fccb8097390 auth_method 0 2023-09-30T01:59:45.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbda19700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:45.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbda19700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:45.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbda19700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:45.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbe21a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:45.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbe21a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:45.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbe21a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:45.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbda19700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:45.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbda19700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:45.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbda19700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:45.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbea1b700 1 --2- 172.21.15.7:0/4044195332 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fccb80a29e0 0x7fccb809ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:45.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbe21a700 10 monclient(hunting): handle_auth_done global_id 4719 payload 824 2023-09-30T01:59:45.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbe21a700 1 -- 172.21.15.7:0/4044195332 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fccb8097390 msgr2=0x7fccb812d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:45.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbe21a700 1 --2- 172.21.15.7:0/4044195332 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fccb8097390 0x7fccb812d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:45.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbe21a700 1 -- 172.21.15.7:0/4044195332 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fccb80a29e0 msgr2=0x7fccb809ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:45.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbe21a700 1 --2- 172.21.15.7:0/4044195332 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fccb80a29e0 0x7fccb809ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:45.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbe21a700 10 monclient: _finish_hunting 0 2023-09-30T01:59:45.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbe21a700 1 monclient: found mon.noname-a 2023-09-30T01:59:45.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbe21a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:45.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbe21a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:45.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbe21a700 1 -- 172.21.15.7:0/4044195332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb0009420 con 0x7fccb8096f90 2023-09-30T01:59:45.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbea1b700 10 monclient: get_auth_request con 0x7fccb80a29e0 auth_method 0 2023-09-30T01:59:45.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbda19700 1 --2- 172.21.15.7:0/4044195332 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fccb8097390 0x7fccb812d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:45.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.892+0000 7fccbea1b700 1 --2- 172.21.15.7:0/4044195332 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fccb80a29e0 0x7fccb809ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:59:45.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccbe21a700 1 -- 172.21.15.7:0/4044195332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccb8096f90 msgr2=0x7fccb8141480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:45.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccbe21a700 1 --2- 172.21.15.7:0/4044195332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccb8096f90 0x7fccb8141480 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fccb4009fd0 tx=0x7fccb4009970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:45.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccae7fc700 1 -- 172.21.15.7:0/4044195332 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb4014420 con 0x7fccb8096f90 2023-09-30T01:59:45.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:45.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccae7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:45.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccae7fc700 10 monclient: dump: 2023-09-30T01:59:45.909 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:45.909 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:45.909 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:45.910 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:45.910 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:45.910 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:45.910 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:45.910 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:45.910 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:45.910 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:45.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccae7fc700 10 monclient: _finish_auth 0 2023-09-30T01:59:45.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccae7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:45.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccc047e700 5 monclient: authenticate success, global_id 4719 2023-09-30T01:59:45.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccae7fc700 1 -- 172.21.15.7:0/4044195332 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb4014580 con 0x7fccb8096f90 2023-09-30T01:59:45.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccc047e700 10 monclient: _renew_subs 2023-09-30T01:59:45.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccc047e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:45.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccc047e700 1 -- 172.21.15.7:0/4044195332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb809eb20 con 0x7fccb8096f90 2023-09-30T01:59:45.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccae7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:45.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccae7fc700 1 -- 172.21.15.7:0/4044195332 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb4014920 con 0x7fccb8096f90 2023-09-30T01:59:45.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccae7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:45.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccae7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:45.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccbca17700 4 set_mon_vals no callback set 2023-09-30T01:59:45.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccae7fc700 10 monclient: dump: 2023-09-30T01:59:45.912 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:45.912 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:45.912 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:45.912 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:45.912 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:45.912 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:45.912 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:45.912 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:45.912 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:45.912 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:45.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccc047e700 10 monclient: _renew_subs 2023-09-30T01:59:45.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccc047e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:45.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.894+0000 7fccc047e700 1 -- 172.21.15.7:0/4044195332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb8142250 con 0x7fccb8096f90 2023-09-30T01:59:45.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.895+0000 7fccae7fc700 1 -- 172.21.15.7:0/4044195332 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fccb4014d10 con 0x7fccb8096f90 2023-09-30T01:59:45.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.895+0000 7fccc047e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:45.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.895+0000 7fccc047e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:45.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.895+0000 7fccc047e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:45.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.895+0000 7fccc047e700 1 -- 172.21.15.7:0/4044195332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb8064c10 con 0x7fccb8096f90 2023-09-30T01:59:45.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.895+0000 7fccae7fc700 1 --2- 172.21.15.7:0/4044195332 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fcc9c0357e0 0x7fcc9c037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:45.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.895+0000 7fccae7fc700 1 -- 172.21.15.7:0/4044195332 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fccb4048490 con 0x7fccb8096f90 2023-09-30T01:59:45.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.896+0000 7fccbda19700 1 --2- 172.21.15.7:0/4044195332 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fcc9c0357e0 0x7fcc9c037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:45.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.896+0000 7fccbda19700 10 monclient: get_auth_request con 0x7fcc9c0357e0 auth_method 0 2023-09-30T01:59:45.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.898+0000 7fccbda19700 1 --2- 172.21.15.7:0/4044195332 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fcc9c0357e0 0x7fcc9c037c90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fccb809ef50 tx=0x7fcca8007f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:45.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.900+0000 7fccae7fc700 1 -- 172.21.15.7:0/4044195332 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fccb401be60 con 0x7fccb8096f90 2023-09-30T01:59:45.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.900+0000 7fccae7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:45.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:45.900+0000 7fccae7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:46.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:46.135+0000 7fccc047e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:59:46.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:46.135+0000 7fccc047e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:46.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:46.135+0000 7fccc047e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:46.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:46.135+0000 7fccc047e700 1 -- 172.21.15.7:0/4044195332 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fccb8099510 con 0x7fccb8096f90 2023-09-30T01:59:46.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:46.135+0000 7fccae7fc700 1 -- 172.21.15.7:0/4044195332 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fccb401ba70 con 0x7fccb8096f90 2023-09-30T01:59:46.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:46.135+0000 7fccae7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:46.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:46.135+0000 7fccae7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:59:46.136 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:59:46.136 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:46.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:46.137+0000 7fccc047e700 1 -- 172.21.15.7:0/4044195332 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fcc9c0357e0 msgr2=0x7fcc9c037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:46.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:46.137+0000 7fccc047e700 1 --2- 172.21.15.7:0/4044195332 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fcc9c0357e0 0x7fcc9c037c90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fccb809ef50 tx=0x7fcca8007f90 comp rx=0 tx=0).stop 2023-09-30T01:59:46.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:46.137+0000 7fccc047e700 10 monclient: shutdown 2023-09-30T01:59:46.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:46.137+0000 7fccc047e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:46.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:46.137+0000 7fccc047e700 1 -- 172.21.15.7:0/4044195332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccb8096f90 msgr2=0x7fccb8141480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:46.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:46.137+0000 7fccc047e700 1 --2- 172.21.15.7:0/4044195332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccb8096f90 0x7fccb8141480 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fccb4009fd0 tx=0x7fccb4009970 comp rx=0 tx=0).stop 2023-09-30T01:59:46.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:46.137+0000 7fccc047e700 1 -- 172.21.15.7:0/4044195332 shutdown_connections 2023-09-30T01:59:46.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:46.137+0000 7fccc047e700 1 --2- 172.21.15.7:0/4044195332 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fcc9c0357e0 0x7fcc9c037c90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:46.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:46.137+0000 7fccc047e700 1 --2- 172.21.15.7:0/4044195332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fccb8096f90 0x7fccb8141480 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:46.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:46.137+0000 7fccc047e700 1 --2- 172.21.15.7:0/4044195332 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fccb80a29e0 0x7fccb809ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:46.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:46.137+0000 7fccc047e700 1 --2- 172.21.15.7:0/4044195332 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fccb8097390 0x7fccb812d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:46.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:46.137+0000 7fccc047e700 1 -- 172.21.15.7:0/4044195332 >> 172.21.15.7:0/4044195332 conn(0x7fccb80710d0 msgr2=0x7fccb81307e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:46.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:46.137+0000 7fccc047e700 1 -- 172.21.15.7:0/4044195332 shutdown_connections 2023-09-30T01:59:46.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:46.137+0000 7fccc047e700 1 -- 172.21.15.7:0/4044195332 wait complete. 2023-09-30T01:59:49.150 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:49.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.278+0000 7f62833ea700 10 monclient: get_monmap_and_config 2023-09-30T01:59:49.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.278+0000 7f62833ea700 10 monclient: build_initial_monmap 2023-09-30T01:59:49.278 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.278+0000 7f62833ea700 10 monclient: monmap: 2023-09-30T01:59:49.278 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:49.279481+0000 2023-09-30T01:59:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:49.279481+0000 2023-09-30T01:59:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:49.279 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:49.279 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:49.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.278+0000 7f62833ea700 1 Processor -- start 2023-09-30T01:59:49.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.279+0000 7f62833ea700 1 -- start start 2023-09-30T01:59:49.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.279+0000 7f62833ea700 10 monclient: init 2023-09-30T01:59:49.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.279+0000 7f62833ea700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:49.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.279+0000 7f62833ea700 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-09-30T01:59:49.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.279+0000 7f62833ea700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f627c096a50 addr v1:172.21.15.50:6789/0 2023-09-30T01:59:49.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.279+0000 7f62833ea700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f627c096e20 0x7f627c097240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:49.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.279+0000 7f62833ea700 10 monclient(hunting): picked mon.noname-c con 0x7f627c096e20 addr v2:172.21.15.50:3300/0 2023-09-30T01:59:49.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.279+0000 7f62833ea700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f627c1376f0 addr v1:172.21.15.7:6789/0 2023-09-30T01:59:49.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.279+0000 7f62833ea700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f627c099480 con 0x7f627c1376f0 2023-09-30T01:59:49.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.279+0000 7f62833ea700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f627c097bf0 con 0x7f627c096a50 2023-09-30T01:59:49.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.279+0000 7f62833ea700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:49.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.279+0000 7f62833ea700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f627c132ef0 con 0x7f627c096e20 2023-09-30T01:59:49.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.279+0000 7f62833ea700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:49.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.279+0000 7f62833ea700 10 monclient(hunting): authenticate will time out at 1478.344795s 2023-09-30T01:59:49.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.279+0000 7f6281987700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f627c1376f0 0x7f627c139ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46782/0 (socket says 172.21.15.7:46782) 2023-09-30T01:59:49.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.279+0000 7f6281987700 1 -- 172.21.15.7:0/1205135398 learned_addr learned my addr 172.21.15.7:0/1205135398 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:59:49.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.280+0000 7f6280985700 1 --2- 172.21.15.7:0/1205135398 >> v2:172.21.15.50:3300/0 conn(0x7f627c096e20 0x7f627c097240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:49.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.280+0000 7f6280985700 10 monclient(hunting): get_auth_request con 0x7f627c096e20 auth_method 0 2023-09-30T01:59:49.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.280+0000 7f6280985700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:49.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.280+0000 7f6280985700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:49.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.280+0000 7f6280985700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:49.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.280+0000 7f6280985700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:49.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.280+0000 7f6280985700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:49.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.280+0000 7f6280985700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:49.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.280+0000 7f6280985700 10 monclient(hunting): handle_auth_done global_id 4597 payload 274 2023-09-30T01:59:49.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.280+0000 7f6280985700 1 -- 172.21.15.7:0/1205135398 >> v1:172.21.15.50:6789/0 conn(0x7f627c096a50 legacy=0x7f627c132750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:49.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.280+0000 7f6280985700 1 -- 172.21.15.7:0/1205135398 >> v1:172.21.15.7:6789/0 conn(0x7f627c1376f0 legacy=0x7f627c139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:49.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.280+0000 7f6280985700 10 monclient: _finish_hunting 0 2023-09-30T01:59:49.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.280+0000 7f6280985700 1 monclient: found mon.noname-c 2023-09-30T01:59:49.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.280+0000 7f6280985700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:49.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.280+0000 7f6280985700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3300/0 2023-09-30T01:59:49.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.280+0000 7f6280985700 1 -- 172.21.15.7:0/1205135398 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627c0991a0 con 0x7f627c096e20 2023-09-30T01:59:49.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.282+0000 7f6280985700 1 -- 172.21.15.7:0/1205135398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f627c096e20 msgr2=0x7f627c097240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:49.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.282+0000 7f6280985700 1 --2- 172.21.15.7:0/1205135398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f627c096e20 0x7f627c097240 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f62700098e0 tx=0x7f62700091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d166b99e98b12a99 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:49.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f627bfff700 1 -- 172.21.15.7:0/1205135398 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627000b9f0 con 0x7f627c096e20 2023-09-30T01:59:49.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f627bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:49.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f627bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:59:49.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f627bfff700 10 monclient: dump: 2023-09-30T01:59:49.291 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:49.291 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:49.291 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:49.291 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:49.291 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:49.291 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:49.291 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:49.291 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:49.291 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:49.291 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:49.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f627bfff700 10 monclient: _finish_auth 0 2023-09-30T01:59:49.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f627bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:49.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f627bfff700 1 -- 172.21.15.7:0/1205135398 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627000bb50 con 0x7f627c096e20 2023-09-30T01:59:49.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f62833ea700 5 monclient: authenticate success, global_id 4597 2023-09-30T01:59:49.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f62833ea700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:49.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f627bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:49.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f627bfff700 1 -- 172.21.15.7:0/1205135398 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6270011420 con 0x7f627c096e20 2023-09-30T01:59:49.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f62833ea700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:49.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f62833ea700 4 set_mon_vals no callback set 2023-09-30T01:59:49.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f627bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:49.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f627bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:59:49.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f627bfff700 10 monclient: dump: 2023-09-30T01:59:49.293 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:49.293 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:49.293 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:49.293 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:49.293 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:49.293 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:49.293 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:49.294 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:49.294 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:49.294 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:49.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f62833ea700 10 monclient: shutdown 2023-09-30T01:59:49.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f62833ea700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:49.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f62833ea700 1 -- 172.21.15.7:0/1205135398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f627c096e20 msgr2=0x7f627c097240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:49.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f62833ea700 1 --2- 172.21.15.7:0/1205135398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f627c096e20 0x7f627c097240 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f62700098e0 tx=0x7f62700091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:49.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f62833ea700 1 -- 172.21.15.7:0/1205135398 shutdown_connections 2023-09-30T01:59:49.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f62833ea700 1 --2- 172.21.15.7:0/1205135398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f627c096e20 0x7f627c097240 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:49.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f62833ea700 1 -- 172.21.15.7:0/1205135398 >> 172.21.15.7:0/1205135398 conn(0x7f627c070cf0 msgr2=0x7f627c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:49.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f62833ea700 1 -- 172.21.15.7:0/1205135398 shutdown_connections 2023-09-30T01:59:49.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.283+0000 7f62833ea700 1 -- 172.21.15.7:0/1205135398 wait complete. 2023-09-30T01:59:49.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.284+0000 7f62833ea700 10 monclient: build_initial_monmap 2023-09-30T01:59:49.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.284+0000 7f62833ea700 10 monclient: monmap: 2023-09-30T01:59:49.295 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:49.295 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:49.295 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:49.295 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:49.295 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:49.295 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:49.295 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:49.295 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:49.295 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:49.295 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.284+0000 7f62833ea700 1 Processor -- start 2023-09-30T01:59:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.284+0000 7f62833ea700 1 -- start start 2023-09-30T01:59:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.284+0000 7f62833ea700 10 monclient: init 2023-09-30T01:59:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f62833ea700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f62833ea700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:59:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f62833ea700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f627c096a50 0x7f627c0a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f62833ea700 10 monclient(hunting): picked mon.noname-b con 0x7f627c096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f62833ea700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f627c096e20 0x7f627c141bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f62833ea700 10 monclient(hunting): picked mon.noname-c con 0x7f627c096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f62833ea700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f627c1376f0 0x7f627c142110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f62833ea700 10 monclient(hunting): picked mon.noname-a con 0x7f627c1376f0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:49.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f62833ea700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f62833ea700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627c09e9b0 con 0x7f627c1376f0 2023-09-30T01:59:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f62833ea700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f62833ea700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627c132080 con 0x7f627c096a50 2023-09-30T01:59:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f62833ea700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f62833ea700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f627c0a06c0 con 0x7f627c096e20 2023-09-30T01:59:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f62833ea700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f62833ea700 10 monclient(hunting): authenticate will time out at 1478.350276s 2023-09-30T01:59:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6280985700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f627c096e20 0x7f627c141bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281987700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f627c1376f0 0x7f627c142110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6280985700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f627c096e20 0x7f627c141bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:40552/0 (socket says 172.21.15.7:40552) 2023-09-30T01:59:49.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281987700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f627c1376f0 0x7f627c142110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54390/0 (socket says 172.21.15.7:54390) 2023-09-30T01:59:49.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6280985700 1 -- 172.21.15.7:0/2444644666 learned_addr learned my addr 172.21.15.7:0/2444644666 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:49.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6280985700 10 monclient(hunting): get_auth_request con 0x7f627c096e20 auth_method 0 2023-09-30T01:59:49.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6280985700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:49.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6280985700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:49.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6280985700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:49.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281987700 10 monclient(hunting): get_auth_request con 0x7f627c1376f0 auth_method 0 2023-09-30T01:59:49.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281987700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:49.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281987700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:49.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281987700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:49.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281987700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:49.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281987700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:49.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281987700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:49.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6280985700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:49.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6280985700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:49.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6280985700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:49.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281186700 1 --2- 172.21.15.7:0/2444644666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f627c096a50 0x7f627c0a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:49.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281186700 10 monclient(hunting): get_auth_request con 0x7f627c096a50 auth_method 0 2023-09-30T01:59:49.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281186700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:49.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281186700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:49.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281186700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:49.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281987700 10 monclient(hunting): handle_auth_done global_id 4722 payload 824 2023-09-30T01:59:49.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281987700 1 -- 172.21.15.7:0/2444644666 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f627c096e20 msgr2=0x7f627c141bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:49.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281987700 1 --2- 172.21.15.7:0/2444644666 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f627c096e20 0x7f627c141bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:49.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281987700 1 -- 172.21.15.7:0/2444644666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f627c096a50 msgr2=0x7f627c0a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281987700 1 --2- 172.21.15.7:0/2444644666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f627c096a50 0x7f627c0a13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281987700 10 monclient: _finish_hunting 0 2023-09-30T01:59:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281987700 1 monclient: found mon.noname-a 2023-09-30T01:59:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281987700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281987700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6281987700 1 -- 172.21.15.7:0/2444644666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6270009420 con 0x7f627c1376f0 2023-09-30T01:59:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.285+0000 7f6280985700 1 --2- 172.21.15.7:0/2444644666 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f627c096e20 0x7f627c141bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.287+0000 7f6281987700 1 -- 172.21.15.7:0/2444644666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f627c1376f0 msgr2=0x7f627c142110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.287+0000 7f6281987700 1 --2- 172.21.15.7:0/2444644666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f627c1376f0 0x7f627c142110 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f626c00cb60 tx=0x7f626c005340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62797fa700 1 -- 172.21.15.7:0/2444644666 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626c00aac0 con 0x7f627c1376f0 2023-09-30T01:59:49.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62797fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62797fa700 10 monclient: dump: 2023-09-30T01:59:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:49.301 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:49.302 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62797fa700 10 monclient: _finish_auth 0 2023-09-30T01:59:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62797fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62797fa700 1 -- 172.21.15.7:0/2444644666 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f626c00ac40 con 0x7f627c1376f0 2023-09-30T01:59:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62833ea700 5 monclient: authenticate success, global_id 4722 2023-09-30T01:59:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62833ea700 10 monclient: _renew_subs 2023-09-30T01:59:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62833ea700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62833ea700 1 -- 172.21.15.7:0/2444644666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f627c0a03e0 con 0x7f627c1376f0 2023-09-30T01:59:49.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62797fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62797fa700 1 -- 172.21.15.7:0/2444644666 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626c0070f0 con 0x7f627c1376f0 2023-09-30T01:59:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62797fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62797fa700 10 monclient: dump: 2023-09-30T01:59:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:49.303 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:49.304 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f627b7fe700 4 set_mon_vals no callback set 2023-09-30T01:59:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62833ea700 10 monclient: _renew_subs 2023-09-30T01:59:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62833ea700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62833ea700 1 -- 172.21.15.7:0/2444644666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f627c09f6b0 con 0x7f627c1376f0 2023-09-30T01:59:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62833ea700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62833ea700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62797fa700 1 -- 172.21.15.7:0/2444644666 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f626c0074e0 con 0x7f627c1376f0 2023-09-30T01:59:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.288+0000 7f62833ea700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.289+0000 7f62833ea700 1 -- 172.21.15.7:0/2444644666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f627c0a6030 con 0x7f627c1376f0 2023-09-30T01:59:49.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.289+0000 7f62797fa700 1 --2- 172.21.15.7:0/2444644666 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f62680357e0 0x7f6268037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:49.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.289+0000 7f62797fa700 1 -- 172.21.15.7:0/2444644666 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f626c048020 con 0x7f627c1376f0 2023-09-30T01:59:49.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.289+0000 7f6281186700 1 --2- 172.21.15.7:0/2444644666 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f62680357e0 0x7f6268037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:49.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.289+0000 7f6281186700 10 monclient: get_auth_request con 0x7f62680357e0 auth_method 0 2023-09-30T01:59:49.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.291+0000 7f6281186700 1 --2- 172.21.15.7:0/2444644666 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f62680357e0 0x7f6268037c90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6274003870 tx=0x7f6274009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:49.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.293+0000 7f62797fa700 1 -- 172.21.15.7:0/2444644666 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f626c0795a0 con 0x7f627c1376f0 2023-09-30T01:59:49.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.294+0000 7f62797fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:49.306 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.294+0000 7f62797fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:49.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.531+0000 7f62833ea700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:59:49.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.531+0000 7f62833ea700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:49.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.531+0000 7f62833ea700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:49.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.531+0000 7f62833ea700 1 -- 172.21.15.7:0/2444644666 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f627c064c10 con 0x7f627c1376f0 2023-09-30T01:59:49.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.531+0000 7f62797fa700 1 -- 172.21.15.7:0/2444644666 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f626c010950 con 0x7f627c1376f0 2023-09-30T01:59:49.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.531+0000 7f62797fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:49.531 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.531+0000 7f62797fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:59:49.532 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:59:49.532 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:49.533 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.533+0000 7f62833ea700 1 -- 172.21.15.7:0/2444644666 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f62680357e0 msgr2=0x7f6268037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:49.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.533+0000 7f62833ea700 1 --2- 172.21.15.7:0/2444644666 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f62680357e0 0x7f6268037c90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6274003870 tx=0x7f6274009040 comp rx=0 tx=0).stop 2023-09-30T01:59:49.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.533+0000 7f62833ea700 10 monclient: shutdown 2023-09-30T01:59:49.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.533+0000 7f62833ea700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:49.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.533+0000 7f62833ea700 1 -- 172.21.15.7:0/2444644666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f627c1376f0 msgr2=0x7f627c142110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:49.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.533+0000 7f62833ea700 1 --2- 172.21.15.7:0/2444644666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f627c1376f0 0x7f627c142110 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f626c00cb60 tx=0x7f626c005340 comp rx=0 tx=0).stop 2023-09-30T01:59:49.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.533+0000 7f62833ea700 1 -- 172.21.15.7:0/2444644666 shutdown_connections 2023-09-30T01:59:49.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.533+0000 7f62833ea700 1 --2- 172.21.15.7:0/2444644666 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f62680357e0 0x7f6268037c90 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:49.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.533+0000 7f62833ea700 1 --2- 172.21.15.7:0/2444644666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f627c1376f0 0x7f627c142110 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:49.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.533+0000 7f62833ea700 1 --2- 172.21.15.7:0/2444644666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f627c096a50 0x7f627c0a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:49.534 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.533+0000 7f62833ea700 1 --2- 172.21.15.7:0/2444644666 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f627c096e20 0x7f627c141bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:49.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.533+0000 7f62833ea700 1 -- 172.21.15.7:0/2444644666 >> 172.21.15.7:0/2444644666 conn(0x7f627c070cf0 msgr2=0x7f627c12f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:49.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.533+0000 7f62833ea700 1 -- 172.21.15.7:0/2444644666 shutdown_connections 2023-09-30T01:59:49.535 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:49.533+0000 7f62833ea700 1 -- 172.21.15.7:0/2444644666 wait complete. 2023-09-30T01:59:52.545 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T01:59:52.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.676+0000 7f8261a12700 10 monclient: get_monmap_and_config 2023-09-30T01:59:52.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.676+0000 7f8261a12700 10 monclient: build_initial_monmap 2023-09-30T01:59:52.680 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.677+0000 7f8261a12700 10 monclient: monmap: 2023-09-30T01:59:52.680 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:52.680 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:52.680 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:52.678031+0000 2023-09-30T01:59:52.681 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:52.678031+0000 2023-09-30T01:59:52.681 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:52.681 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:52.681 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:52.681 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:52.681 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:52.681 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:52.681 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:52.681 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:52.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.677+0000 7f8261a12700 1 Processor -- start 2023-09-30T01:59:52.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.677+0000 7f8261a12700 1 -- start start 2023-09-30T01:59:52.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.677+0000 7f8261a12700 10 monclient: init 2023-09-30T01:59:52.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f8261a12700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:52.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f8261a12700 10 monclient: _add_conns ranks=[3,4,2,0,1] 2023-09-30T01:59:52.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f8261a12700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f825c096a50 0x7f825c132680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:52.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f8261a12700 10 monclient(hunting): picked mon.noname-a con 0x7f825c096a50 addr v2:172.21.15.7:3300/0 2023-09-30T01:59:52.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f8261a12700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f825c096e20 0x7f825c097220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:52.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f8261a12700 10 monclient(hunting): picked mon.noname-c con 0x7f825c096e20 addr v2:172.21.15.50:3300/0 2023-09-30T01:59:52.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f8261a12700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f825c12ef00 0x7f825c12f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:52.682 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f8261a12700 10 monclient(hunting): picked mon.noname-b con 0x7f825c12ef00 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:52.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f8261a12700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:52.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f8261a12700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f825c099440 con 0x7f825c096a50 2023-09-30T01:59:52.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f8261a12700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:52.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f8261a12700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f825c097b40 con 0x7f825c096e20 2023-09-30T01:59:52.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f8261a12700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:52.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f8261a12700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f825c099160 con 0x7f825c12ef00 2023-09-30T01:59:52.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f8261a12700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:52.683 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f8261a12700 10 monclient(hunting): authenticate will time out at 1481.743138s 2023-09-30T01:59:52.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f825c096a50 0x7f825c132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:52.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825b7fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f825c12ef00 0x7f825c12f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:52.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825b7fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f825c12ef00 0x7f825c12f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:40558/0 (socket says 172.21.15.7:40558) 2023-09-30T01:59:52.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825affd700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f825c096a50 0x7f825c132680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54406/0 (socket says 172.21.15.7:54406) 2023-09-30T01:59:52.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825b7fe700 1 -- 172.21.15.7:0/4158135399 learned_addr learned my addr 172.21.15.7:0/4158135399 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:52.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825b7fe700 10 monclient(hunting): get_auth_request con 0x7f825c12ef00 auth_method 0 2023-09-30T01:59:52.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:52.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:52.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:52.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825affd700 10 monclient(hunting): get_auth_request con 0x7f825c096a50 auth_method 0 2023-09-30T01:59:52.684 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:52.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:52.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:52.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:52.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:52.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:52.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:52.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:52.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:52.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f82527fc700 1 --2- 172.21.15.7:0/4158135399 >> v2:172.21.15.50:3300/0 conn(0x7f825c096e20 0x7f825c097220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:52.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f82527fc700 10 monclient(hunting): get_auth_request con 0x7f825c096e20 auth_method 0 2023-09-30T01:59:52.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f82527fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:52.685 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f82527fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:52.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f82527fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:52.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825b7fe700 10 monclient(hunting): handle_auth_done global_id 4745 payload 274 2023-09-30T01:59:52.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825b7fe700 1 -- 172.21.15.7:0/4158135399 >> v2:172.21.15.50:3300/0 conn(0x7f825c096e20 msgr2=0x7f825c097220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:52.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825b7fe700 1 --2- 172.21.15.7:0/4158135399 >> v2:172.21.15.50:3300/0 conn(0x7f825c096e20 0x7f825c097220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:52.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825b7fe700 1 -- 172.21.15.7:0/4158135399 >> v2:172.21.15.7:3300/0 conn(0x7f825c096a50 msgr2=0x7f825c132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:52.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825b7fe700 1 --2- 172.21.15.7:0/4158135399 >> v2:172.21.15.7:3300/0 conn(0x7f825c096a50 0x7f825c132680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:52.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825b7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:59:52.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825b7fe700 1 monclient: found mon.noname-b 2023-09-30T01:59:52.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:52.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:52.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825b7fe700 1 -- 172.21.15.7:0/4158135399 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825c099720 con 0x7f825c12ef00 2023-09-30T01:59:52.686 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.678+0000 7f825affd700 1 --2- 172.21.15.7:0/4158135399 >> v2:172.21.15.7:3300/0 conn(0x7f825c096a50 0x7f825c132680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:52.692 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.680+0000 7f825b7fe700 1 -- 172.21.15.7:0/4158135399 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f825c12ef00 msgr2=0x7f825c12f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:52.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.680+0000 7f825b7fe700 1 --2- 172.21.15.7:0/4158135399 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f825c12ef00 0x7f825c12f3a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f825c0a5740 tx=0x7f8254009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66e0074060798a44 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:52.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f825a7fc700 1 -- 172.21.15.7:0/4158135399 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825400e070 con 0x7f825c12ef00 2023-09-30T01:59:52.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f825a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:52.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f825a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:52.693 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f825a7fc700 10 monclient: dump: 2023-09-30T01:59:52.693 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:52.693 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:52.693 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:52.693 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:52.694 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:52.694 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:52.694 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:52.694 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:52.695 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:52.695 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:52.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f825a7fc700 10 monclient: _finish_auth 0 2023-09-30T01:59:52.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f825a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:52.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f825a7fc700 1 -- 172.21.15.7:0/4158135399 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825400bc30 con 0x7f825c12ef00 2023-09-30T01:59:52.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f8261a12700 5 monclient: authenticate success, global_id 4745 2023-09-30T01:59:52.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f8261a12700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:52.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f825a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:52.695 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f825a7fc700 1 -- 172.21.15.7:0/4158135399 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82540092e0 con 0x7f825c12ef00 2023-09-30T01:59:52.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f8261a12700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:52.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f8261a12700 4 set_mon_vals no callback set 2023-09-30T01:59:52.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f825a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:52.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f825a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:52.696 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f825a7fc700 10 monclient: dump: 2023-09-30T01:59:52.696 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:52.696 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:52.696 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:52.696 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:52.696 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:52.697 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:52.697 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:52.697 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:52.697 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:52.697 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:52.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f8261a12700 10 monclient: shutdown 2023-09-30T01:59:52.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f8261a12700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:52.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f8261a12700 1 -- 172.21.15.7:0/4158135399 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f825c12ef00 msgr2=0x7f825c12f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:52.697 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f8261a12700 1 --2- 172.21.15.7:0/4158135399 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f825c12ef00 0x7f825c12f3a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f825c0a5740 tx=0x7f8254009840 comp rx=0 tx=0).stop 2023-09-30T01:59:52.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f8261a12700 1 -- 172.21.15.7:0/4158135399 shutdown_connections 2023-09-30T01:59:52.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f8261a12700 1 --2- 172.21.15.7:0/4158135399 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f825c12ef00 0x7f825c12f3a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:52.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f8261a12700 1 --2- 172.21.15.7:0/4158135399 >> v2:172.21.15.7:3300/0 conn(0x7f825c096a50 0x7f825c132680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:52.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f8261a12700 1 --2- 172.21.15.7:0/4158135399 >> v2:172.21.15.50:3300/0 conn(0x7f825c096e20 0x7f825c097220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:52.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f8261a12700 1 -- 172.21.15.7:0/4158135399 >> 172.21.15.7:0/4158135399 conn(0x7f825c070cf0 msgr2=0x7f825c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:52.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f8261a12700 1 -- 172.21.15.7:0/4158135399 shutdown_connections 2023-09-30T01:59:52.698 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.681+0000 7f8261a12700 1 -- 172.21.15.7:0/4158135399 wait complete. 2023-09-30T01:59:52.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.682+0000 7f8261a12700 10 monclient: build_initial_monmap 2023-09-30T01:59:52.699 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.682+0000 7f8261a12700 10 monclient: monmap: 2023-09-30T01:59:52.699 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:52.699 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:52.699 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:52.699 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:52.699 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:52.699 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:52.700 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:52.700 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:52.700 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:52.700 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:52.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.682+0000 7f8261a12700 1 Processor -- start 2023-09-30T01:59:52.700 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.682+0000 7f8261a12700 1 -- start start 2023-09-30T01:59:52.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.682+0000 7f8261a12700 10 monclient: init 2023-09-30T01:59:52.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f8261a12700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:52.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f8261a12700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:59:52.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f8261a12700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f825c096a50 0x7f825c12ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:52.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f8261a12700 10 monclient(hunting): picked mon.noname-b con 0x7f825c096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:52.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f8261a12700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f825c096e20 0x7f825c0a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:52.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f8261a12700 10 monclient(hunting): picked mon.noname-c con 0x7f825c096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:52.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f8261a12700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f825c12ef00 0x7f825c0a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:52.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f8261a12700 10 monclient(hunting): picked mon.noname-a con 0x7f825c12ef00 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:52.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f8261a12700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:52.701 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f8261a12700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825c131de0 con 0x7f825c12ef00 2023-09-30T01:59:52.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f8261a12700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:52.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f8261a12700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825c11e2a0 con 0x7f825c096a50 2023-09-30T01:59:52.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f8261a12700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:52.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f8261a12700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f825c11d090 con 0x7f825c096e20 2023-09-30T01:59:52.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f8261a12700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:52.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f8261a12700 10 monclient(hunting): authenticate will time out at 1481.748203s 2023-09-30T01:59:52.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f825c12ef00 0x7f825c0a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:52.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f82527fc700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f825c096e20 0x7f825c0a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:52.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825b7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f825c12ef00 0x7f825c0a0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54408/0 (socket says 172.21.15.7:54408) 2023-09-30T01:59:52.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825b7fe700 1 -- 172.21.15.7:0/3489987517 learned_addr learned my addr 172.21.15.7:0/3489987517 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:52.702 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825b7fe700 10 monclient(hunting): get_auth_request con 0x7f825c12ef00 auth_method 0 2023-09-30T01:59:52.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:52.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:52.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f82527fc700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f825c096e20 0x7f825c0a0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:40560/0 (socket says 172.21.15.7:40560) 2023-09-30T01:59:52.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:52.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f82527fc700 10 monclient(hunting): get_auth_request con 0x7f825c096e20 auth_method 0 2023-09-30T01:59:52.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f82527fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:52.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f82527fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:52.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f82527fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:52.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825affd700 1 --2- 172.21.15.7:0/3489987517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f825c096a50 0x7f825c12ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:52.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:52.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:52.703 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:52.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f82527fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:52.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f82527fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:52.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f82527fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:52.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825affd700 10 monclient(hunting): get_auth_request con 0x7f825c096a50 auth_method 0 2023-09-30T01:59:52.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:52.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:52.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:52.704 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:52.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:52.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:52.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825b7fe700 10 monclient(hunting): handle_auth_done global_id 4728 payload 824 2023-09-30T01:59:52.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825b7fe700 1 -- 172.21.15.7:0/3489987517 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f825c096e20 msgr2=0x7f825c0a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:52.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825b7fe700 1 --2- 172.21.15.7:0/3489987517 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f825c096e20 0x7f825c0a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:52.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825b7fe700 1 -- 172.21.15.7:0/3489987517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f825c096a50 msgr2=0x7f825c12ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:52.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825b7fe700 1 --2- 172.21.15.7:0/3489987517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f825c096a50 0x7f825c12ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:52.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825b7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:59:52.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825b7fe700 1 monclient: found mon.noname-a 2023-09-30T01:59:52.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:52.705 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:52.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f825b7fe700 1 -- 172.21.15.7:0/3489987517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824c009420 con 0x7f825c12ef00 2023-09-30T01:59:52.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.683+0000 7f82527fc700 1 --2- 172.21.15.7:0/3489987517 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f825c096e20 0x7f825c0a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:52.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.685+0000 7f825b7fe700 1 -- 172.21.15.7:0/3489987517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f825c12ef00 msgr2=0x7f825c0a0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:52.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.685+0000 7f825b7fe700 1 --2- 172.21.15.7:0/3489987517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f825c12ef00 0x7f825c0a0e70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8254003390 tx=0x7f8254018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:52.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.686+0000 7f82537fe700 1 -- 172.21.15.7:0/3489987517 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8254007bb0 con 0x7f825c12ef00 2023-09-30T01:59:52.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.686+0000 7f82537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:52.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.686+0000 7f82537fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:52.706 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.686+0000 7f82537fe700 10 monclient: dump: 2023-09-30T01:59:52.706 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:52.706 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:52.707 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:52.707 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:52.707 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:52.707 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:52.707 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:52.707 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:52.707 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:52.707 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:52.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.686+0000 7f82537fe700 10 monclient: _finish_auth 0 2023-09-30T01:59:52.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.686+0000 7f82537fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:52.707 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.686+0000 7f82537fe700 1 -- 172.21.15.7:0/3489987517 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8254007d10 con 0x7f825c12ef00 2023-09-30T01:59:52.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.686+0000 7f8261a12700 5 monclient: authenticate success, global_id 4728 2023-09-30T01:59:52.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.686+0000 7f8261a12700 10 monclient: _renew_subs 2023-09-30T01:59:52.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.686+0000 7f8261a12700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:52.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.686+0000 7f8261a12700 1 -- 172.21.15.7:0/3489987517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8254009ae0 con 0x7f825c12ef00 2023-09-30T01:59:52.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.686+0000 7f82537fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:52.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.686+0000 7f82537fe700 1 -- 172.21.15.7:0/3489987517 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82540266b0 con 0x7f825c12ef00 2023-09-30T01:59:52.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.686+0000 7f82537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:52.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.686+0000 7f82537fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:52.708 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.686+0000 7f82537fe700 10 monclient: dump: 2023-09-30T01:59:52.708 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:52.708 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:52.709 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:52.709 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:52.709 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:52.709 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:52.709 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:52.709 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:52.709 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:52.709 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:52.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.686+0000 7f8261a12700 10 monclient: _renew_subs 2023-09-30T01:59:52.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.686+0000 7f8261a12700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:52.709 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.686+0000 7f8261a12700 1 -- 172.21.15.7:0/3489987517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f825c11c3d0 con 0x7f825c12ef00 2023-09-30T01:59:52.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.686+0000 7f8259ffb700 4 set_mon_vals no callback set 2023-09-30T01:59:52.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.687+0000 7f8261a12700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:52.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.687+0000 7f8261a12700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:52.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.687+0000 7f8261a12700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:52.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.687+0000 7f8261a12700 1 -- 172.21.15.7:0/3489987517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f825c11e420 con 0x7f825c12ef00 2023-09-30T01:59:52.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.687+0000 7f825b7fe700 1 --2- 172.21.15.7:0/3489987517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f825c12ef00 0x7f825c0a0e70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8254003390 tx=0x7f8254018000 comp rx=0 tx=0).queue_received will delay after 1.69604e+09 on 0x7f8254026b20 mgrmap(e 5) v1 2023-09-30T01:59:52.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.687+0000 7f82537fe700 1 -- 172.21.15.7:0/3489987517 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f8254026b20 con 0x7f825c12ef00 2023-09-30T01:59:52.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.687+0000 7f82537fe700 1 --2- 172.21.15.7:0/3489987517 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f823c0357e0 0x7f823c037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:52.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.687+0000 7f825affd700 1 --2- 172.21.15.7:0/3489987517 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f823c0357e0 0x7f823c037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:52.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.687+0000 7f825affd700 10 monclient: get_auth_request con 0x7f823c0357e0 auth_method 0 2023-09-30T01:59:52.710 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.690+0000 7f825affd700 1 --2- 172.21.15.7:0/3489987517 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f823c0357e0 0x7f823c037c90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f824c009ea0 tx=0x7f824c0091a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:52.711 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:52.692+0000 7f82537fe700 1 -- 172.21.15.7:0/3489987517 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f8254049a00 con 0x7f825c12ef00 2023-09-30T01:59:53.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.515+0000 7f82537fe700 1 -- 172.21.15.7:0/3489987517 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f825c11e420 con 0x7f825c12ef00 2023-09-30T01:59:53.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.515+0000 7f82537fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:53.515 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.515+0000 7f82537fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:53.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.756+0000 7f8261a12700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T01:59:53.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.756+0000 7f8261a12700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:53.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.756+0000 7f8261a12700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:53.756 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.756+0000 7f8261a12700 1 -- 172.21.15.7:0/3489987517 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f825c11e600 con 0x7f825c12ef00 2023-09-30T01:59:53.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.756+0000 7f82537fe700 1 -- 172.21.15.7:0/3489987517 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f825c11e600 con 0x7f825c12ef00 2023-09-30T01:59:53.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.756+0000 7f82537fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T01:59:53.757 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.756+0000 7f82537fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:59:53.757 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:59:53.757 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T01:59:53.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.758+0000 7f8261a12700 1 -- 172.21.15.7:0/3489987517 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f823c0357e0 msgr2=0x7f823c037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:53.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.758+0000 7f8261a12700 1 --2- 172.21.15.7:0/3489987517 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f823c0357e0 0x7f823c037c90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f824c009ea0 tx=0x7f824c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:53.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.758+0000 7f8261a12700 10 monclient: shutdown 2023-09-30T01:59:53.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.758+0000 7f8261a12700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:53.758 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.758+0000 7f8261a12700 1 -- 172.21.15.7:0/3489987517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f825c12ef00 msgr2=0x7f825c0a0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:53.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.758+0000 7f8261a12700 1 --2- 172.21.15.7:0/3489987517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f825c12ef00 0x7f825c0a0e70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8254003390 tx=0x7f8254018000 comp rx=0 tx=0).stop 2023-09-30T01:59:53.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.758+0000 7f8261a12700 1 -- 172.21.15.7:0/3489987517 shutdown_connections 2023-09-30T01:59:53.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.758+0000 7f8261a12700 1 --2- 172.21.15.7:0/3489987517 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f823c0357e0 0x7f823c037c90 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:53.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.758+0000 7f8261a12700 1 --2- 172.21.15.7:0/3489987517 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f825c12ef00 0x7f825c0a0e70 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:53.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.758+0000 7f8261a12700 1 --2- 172.21.15.7:0/3489987517 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f825c096a50 0x7f825c12ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:53.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.758+0000 7f8261a12700 1 --2- 172.21.15.7:0/3489987517 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f825c096e20 0x7f825c0a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:53.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.758+0000 7f8261a12700 1 -- 172.21.15.7:0/3489987517 >> 172.21.15.7:0/3489987517 conn(0x7f825c070cf0 msgr2=0x7f825c131380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:53.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.758+0000 7f8261a12700 1 -- 172.21.15.7:0/3489987517 shutdown_connections 2023-09-30T01:59:53.759 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.758+0000 7f8261a12700 1 -- 172.21.15.7:0/3489987517 wait complete. 2023-09-30T01:59:53.770 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-09-30T01:59:53.770 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:59:53.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.888+0000 7f242768f700 10 monclient: get_monmap_and_config 2023-09-30T01:59:53.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.888+0000 7f242768f700 10 monclient: build_initial_monmap 2023-09-30T01:59:53.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.888+0000 7f242768f700 10 monclient: monmap: 2023-09-30T01:59:53.889 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:53.889 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:53.889 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:53.889243+0000 2023-09-30T01:59:53.889 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:53.889243+0000 2023-09-30T01:59:53.889 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:53.889 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:53.889 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:53.889 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:53.889 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:53.889 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:53.889 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:53.889 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:53.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.888+0000 7f242768f700 1 Processor -- start 2023-09-30T01:59:53.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.888+0000 7f242768f700 1 -- start start 2023-09-30T01:59:53.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.888+0000 7f242768f700 10 monclient: init 2023-09-30T01:59:53.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242768f700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:53.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242768f700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2023-09-30T01:59:53.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242768f700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2420096a50 0x7f2420132680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:53.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242768f700 10 monclient(hunting): picked mon.noname-b con 0x7f2420096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:53.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242768f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2420096e20 addr v1:172.21.15.50:6789/0 2023-09-30T01:59:53.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242768f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2420137710 addr v1:172.21.15.7:6789/0 2023-09-30T01:59:53.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242768f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2420099440 con 0x7f2420137710 2023-09-30T01:59:53.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242768f700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2420097b40 con 0x7f2420096e20 2023-09-30T01:59:53.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242768f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:53.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242768f700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2420099160 con 0x7f2420096a50 2023-09-30T01:59:53.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242768f700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:53.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242768f700 10 monclient(hunting): authenticate will time out at 1482.954444s 2023-09-30T01:59:53.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242542b700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2420096a50 0x7f2420132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:53.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f2425c2c700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f2420137710 0x7f2420139af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48928/0 (socket says 172.21.15.7:48928) 2023-09-30T01:59:53.899 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f2425c2c700 1 -- 172.21.15.7:0/2387876229 learned_addr learned my addr 172.21.15.7:0/2387876229 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:59:53.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242542b700 10 monclient(hunting): get_auth_request con 0x7f2420096a50 auth_method 0 2023-09-30T01:59:53.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242542b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:53.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242542b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:53.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242542b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:53.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242542b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:53.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242542b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:53.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242542b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:53.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242542b700 10 monclient(hunting): handle_auth_done global_id 4751 payload 274 2023-09-30T01:59:53.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242542b700 1 -- 172.21.15.7:0/2387876229 >> v1:172.21.15.50:6789/0 conn(0x7f2420096e20 legacy=0x7f24200971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:53.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242542b700 1 -- 172.21.15.7:0/2387876229 >> v1:172.21.15.7:6789/0 conn(0x7f2420137710 legacy=0x7f2420139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:53.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242542b700 10 monclient: _finish_hunting 0 2023-09-30T01:59:53.900 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242542b700 1 monclient: found mon.noname-b 2023-09-30T01:59:53.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242542b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:53.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242542b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:53.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.889+0000 7f242542b700 1 -- 172.21.15.7:0/2387876229 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2420099720 con 0x7f2420096a50 2023-09-30T01:59:53.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f242542b700 1 -- 172.21.15.7:0/2387876229 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2420096a50 msgr2=0x7f2420132680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:53.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f242542b700 1 --2- 172.21.15.7:0/2387876229 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2420096a50 0x7f2420132680 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2418009910 tx=0x7f24180091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c350c29848ef60c5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:53.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f241ffff700 1 -- 172.21.15.7:0/2387876229 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241800b9f0 con 0x7f2420096a50 2023-09-30T01:59:53.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f241ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:53.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f241ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:53.901 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f241ffff700 10 monclient: dump: 2023-09-30T01:59:53.901 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:53.901 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:53.901 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:53.902 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:53.902 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:53.902 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:53.902 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:53.902 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:53.902 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:53.902 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:53.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f241ffff700 10 monclient: _finish_auth 0 2023-09-30T01:59:53.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f241ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:53.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f241ffff700 1 -- 172.21.15.7:0/2387876229 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f241800bb50 con 0x7f2420096a50 2023-09-30T01:59:53.902 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f242768f700 5 monclient: authenticate success, global_id 4751 2023-09-30T01:59:53.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f242768f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:53.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f241ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:53.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f242768f700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:53.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f241ffff700 1 -- 172.21.15.7:0/2387876229 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2418011420 con 0x7f2420096a50 2023-09-30T01:59:53.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f241ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:53.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f242768f700 4 set_mon_vals no callback set 2023-09-30T01:59:53.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f241ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:53.903 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f241ffff700 10 monclient: dump: 2023-09-30T01:59:53.903 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:53.903 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:53.903 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:53.903 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:53.904 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:53.904 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:53.904 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:53.904 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:53.904 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:53.904 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:53.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f242768f700 10 monclient: shutdown 2023-09-30T01:59:53.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f242768f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:53.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f242768f700 1 -- 172.21.15.7:0/2387876229 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2420096a50 msgr2=0x7f2420132680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:53.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f242768f700 1 --2- 172.21.15.7:0/2387876229 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2420096a50 0x7f2420132680 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2418009910 tx=0x7f24180091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:53.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f242768f700 1 -- 172.21.15.7:0/2387876229 shutdown_connections 2023-09-30T01:59:53.904 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f242768f700 1 --2- 172.21.15.7:0/2387876229 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2420096a50 0x7f2420132680 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:53.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f242768f700 1 -- 172.21.15.7:0/2387876229 >> 172.21.15.7:0/2387876229 conn(0x7f2420070cf0 msgr2=0x7f24200710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:53.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f242768f700 1 -- 172.21.15.7:0/2387876229 shutdown_connections 2023-09-30T01:59:53.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.892+0000 7f242768f700 1 -- 172.21.15.7:0/2387876229 wait complete. 2023-09-30T01:59:53.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.893+0000 7f242768f700 10 monclient: build_initial_monmap 2023-09-30T01:59:53.905 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.893+0000 7f242768f700 10 monclient: monmap: 2023-09-30T01:59:53.905 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:53.905 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:53.905 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:53.905 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:53.905 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:53.905 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:53.905 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:53.906 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:53.906 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:53.906 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:53.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.893+0000 7f242768f700 1 Processor -- start 2023-09-30T01:59:53.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.893+0000 7f242768f700 1 -- start start 2023-09-30T01:59:53.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.893+0000 7f242768f700 10 monclient: init 2023-09-30T01:59:53.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242768f700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:53.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242768f700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:59:53.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242768f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2420096a50 0x7f24200a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:53.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242768f700 10 monclient(hunting): picked mon.noname-a con 0x7f2420096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:53.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242768f700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2420096e20 0x7f242009cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:53.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242768f700 10 monclient(hunting): picked mon.noname-c con 0x7f2420096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:53.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242768f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2420137710 0x7f242009d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:53.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242768f700 10 monclient(hunting): picked mon.noname-b con 0x7f2420137710 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:53.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242768f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:53.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242768f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24200a0950 con 0x7f2420096a50 2023-09-30T01:59:53.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242768f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:53.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242768f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24201334e0 con 0x7f2420137710 2023-09-30T01:59:53.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242768f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:53.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242768f700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2420131db0 con 0x7f2420096e20 2023-09-30T01:59:53.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242768f700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:53.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242768f700 10 monclient(hunting): authenticate will time out at 1482.959360s 2023-09-30T01:59:53.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242542b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2420096a50 0x7f24200a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:53.907 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f2424c2a700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2420096e20 0x7f242009cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:53.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242542b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2420096a50 0x7f24200a1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49270/0 (socket says 172.21.15.7:49270) 2023-09-30T01:59:53.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242542b700 1 -- 172.21.15.7:0/1711397234 learned_addr learned my addr 172.21.15.7:0/1711397234 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:53.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242542b700 10 monclient(hunting): get_auth_request con 0x7f2420096a50 auth_method 0 2023-09-30T01:59:53.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242542b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:53.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242542b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:53.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242542b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:53.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f2424c2a700 10 monclient(hunting): get_auth_request con 0x7f2420096e20 auth_method 0 2023-09-30T01:59:53.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f2424c2a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:53.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f2424c2a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:53.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f2424c2a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:53.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f2424c2a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:53.908 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f2424c2a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:53.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f2424c2a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:53.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242542b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:53.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242542b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:53.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242542b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:53.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f2425c2c700 1 --2- 172.21.15.7:0/1711397234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2420137710 0x7f242009d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:53.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f2424c2a700 10 monclient(hunting): handle_auth_done global_id 4754 payload 824 2023-09-30T01:59:53.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f2424c2a700 1 -- 172.21.15.7:0/1711397234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2420137710 msgr2=0x7f242009d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:53.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f2424c2a700 1 --2- 172.21.15.7:0/1711397234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2420137710 0x7f242009d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:53.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f2424c2a700 1 -- 172.21.15.7:0/1711397234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2420096a50 msgr2=0x7f24200a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:53.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f2424c2a700 1 --2- 172.21.15.7:0/1711397234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2420096a50 0x7f24200a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:53.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f2424c2a700 10 monclient: _finish_hunting 0 2023-09-30T01:59:53.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f2424c2a700 1 monclient: found mon.noname-c 2023-09-30T01:59:53.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f2424c2a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:53.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f2424c2a700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:59:53.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f2424c2a700 1 -- 172.21.15.7:0/1711397234 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2418009420 con 0x7f2420096e20 2023-09-30T01:59:53.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.894+0000 7f242542b700 1 --2- 172.21.15.7:0/1711397234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2420096a50 0x7f24200a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:53.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.896+0000 7f2424c2a700 1 -- 172.21.15.7:0/1711397234 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2420096e20 msgr2=0x7f242009cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:53.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.896+0000 7f2424c2a700 1 --2- 172.21.15.7:0/1711397234 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2420096e20 0x7f242009cbf0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f241400c540 tx=0x7f2414007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:53.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.897+0000 7f241d7fa700 1 -- 172.21.15.7:0/1711397234 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24140117a0 con 0x7f2420096e20 2023-09-30T01:59:53.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.897+0000 7f241d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:53.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.897+0000 7f241d7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:59:53.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.897+0000 7f241d7fa700 10 monclient: dump: 2023-09-30T01:59:53.911 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:53.911 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:53.911 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:53.911 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:53.911 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:53.911 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:53.911 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:53.911 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:53.912 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:53.912 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:53.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.897+0000 7f241d7fa700 10 monclient: _finish_auth 0 2023-09-30T01:59:53.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.897+0000 7f241d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:53.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.897+0000 7f241d7fa700 1 -- 172.21.15.7:0/1711397234 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2414011920 con 0x7f2420096e20 2023-09-30T01:59:53.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.897+0000 7f242768f700 5 monclient: authenticate success, global_id 4754 2023-09-30T01:59:53.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.897+0000 7f242768f700 10 monclient: _renew_subs 2023-09-30T01:59:53.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.897+0000 7f242768f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:53.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.897+0000 7f242768f700 1 -- 172.21.15.7:0/1711397234 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f242009ed50 con 0x7f2420096e20 2023-09-30T01:59:53.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.897+0000 7f241d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:53.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.897+0000 7f241d7fa700 1 -- 172.21.15.7:0/1711397234 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2414011ce0 con 0x7f2420096e20 2023-09-30T01:59:53.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.897+0000 7f241d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:53.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.897+0000 7f241d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:53.913 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.897+0000 7f241d7fa700 10 monclient: dump: 2023-09-30T01:59:53.913 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:53.913 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:53.913 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:53.913 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:53.913 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:53.913 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:53.913 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:53.913 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:53.913 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:53.914 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:53.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.897+0000 7f242768f700 10 monclient: _renew_subs 2023-09-30T01:59:53.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.897+0000 7f241f7fe700 4 set_mon_vals no callback set 2023-09-30T01:59:53.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.897+0000 7f242768f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:53.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.897+0000 7f242768f700 1 -- 172.21.15.7:0/1711397234 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f242009e7d0 con 0x7f2420096e20 2023-09-30T01:59:53.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.897+0000 7f241d7fa700 1 -- 172.21.15.7:0/1711397234 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f2414012630 con 0x7f2420096e20 2023-09-30T01:59:53.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.898+0000 7f241d7fa700 1 --2- 172.21.15.7:0/1711397234 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f240c0357e0 0x7f240c037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:53.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.898+0000 7f241d7fa700 1 -- 172.21.15.7:0/1711397234 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f2414047990 con 0x7f2420096e20 2023-09-30T01:59:53.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.898+0000 7f242768f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:53.914 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.898+0000 7f242768f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:53.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.898+0000 7f242768f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:53.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.898+0000 7f242768f700 1 -- 172.21.15.7:0/1711397234 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2420064c10 con 0x7f2420096e20 2023-09-30T01:59:53.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.898+0000 7f242542b700 1 --2- 172.21.15.7:0/1711397234 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f240c0357e0 0x7f240c037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:53.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.898+0000 7f242542b700 10 monclient: get_auth_request con 0x7f240c0357e0 auth_method 0 2023-09-30T01:59:53.915 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.900+0000 7f242542b700 1 --2- 172.21.15.7:0/1711397234 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f240c0357e0 0x7f240c037c90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f24180093a0 tx=0x7f2418016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:53.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.903+0000 7f241d7fa700 1 -- 172.21.15.7:0/1711397234 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2414002090 con 0x7f2420096e20 2023-09-30T01:59:53.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.903+0000 7f241d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:53.916 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:53.903+0000 7f241d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:54.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.095+0000 7f242768f700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:59:54.095 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.095+0000 7f242768f700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:59:54.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.095+0000 7f242768f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:54.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.095+0000 7f242768f700 1 -- 172.21.15.7:0/1711397234 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24200a2480 con 0x7f2420096e20 2023-09-30T01:59:54.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.096+0000 7f241d7fa700 1 -- 172.21.15.7:0/1711397234 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+11236 (secure 0 0 0) 0x7f241401b3c0 con 0x7f2420096e20 2023-09-30T01:59:54.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.096+0000 7f241d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:59:54.096 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.096+0000 7f241d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:59:54.097 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:59:54.097 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":15,"fsid":"2ea1044d-d984-49b9-baad-7f36bd454a19","created":"2023-09-30T01:54:34.356209+0000","modified":"2023-09-30T01:54:47.895378+0000","last_up_change":"2023-09-30T01:54:43.359934+0000","last_in_change":"2023-09-30T01:54:36.628886+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T01:54:44.376777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T01:54:45.091042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"e44e9247-cb86-4a16-ac1f-f021cdf5108f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":86601},{"type":"v1","addr":"172.21.15.7:6801","nonce":86601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":86601},{"type":"v1","addr":"172.21.15.7:6803","nonce":86601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":86601},{"type":"v1","addr":"172.21.15.7:6807","nonce":86601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":86601},{"type":"v1","addr":"172.21.15.7:6805","nonce":86601}]},"public_addr":"172.21.15.7:6801/86601","cluster_addr":"172.21.15.7:6803/86601","heartbeat_back_addr":"172.21.15.7:6807/86601","heartbeat_front_addr":"172.21.15.7:6805/86601","state":["exists","up"]},{"osd":1,"uuid":"e30730fd-1e49-44bf-bdc9-087e23727df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":86600},{"type":"v1","addr":"172.21.15.7:6817","nonce":86600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":86600},{"type":"v1","addr":"172.21.15.7:6819","nonce":86600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":86600},{"type":"v1","addr":"172.21.15.7:6823","nonce":86600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":86600},{"type":"v1","addr":"172.21.15.7:6821","nonce":86600}]},"public_addr":"172.21.15.7:6817/86600","cluster_addr":"172.21.15.7:6819/86600","heartbeat_back_addr":"172.21.15.7:6823/86600","heartbeat_front_addr":"172.21.15.7:6821/86600","state":["exists","up"]},{"osd":2,"uuid":"b39c552e-f5cc-4e4f-90a3-5aa5f5127a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":86602},{"type":"v1","addr":"172.21.15.7:6809","nonce":86602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":86602},{"type":"v1","addr":"172.21.15.7:6811","nonce":86602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":86602},{"type":"v1","addr":"172.21.15.7:6815","nonce":86602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":86602},{"type":"v1","addr":"172.21.15.7:6813","nonce":86602}]},"public_addr":"172.21.15.7:6809/86602","cluster_addr":"172.21.15.7:6811/86602","heartbeat_back_addr":"172.21.15.7:6815/86602","heartbeat_front_addr":"172.21.15.7:6813/86602","state":["exists","up"]},{"osd":3,"uuid":"15cd89ae-049a-4e1a-8c6b-7d086ca6e1df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":86633},{"type":"v1","addr":"172.21.15.50:6807","nonce":86633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":86633},{"type":"v1","addr":"172.21.15.50:6812","nonce":86633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6819","nonce":86633},{"type":"v1","addr":"172.21.15.50:6821","nonce":86633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6815","nonce":86633},{"type":"v1","addr":"172.21.15.50:6818","nonce":86633}]},"public_addr":"172.21.15.50:6807/86633","cluster_addr":"172.21.15.50:6812/86633","heartbeat_back_addr":"172.21.15.50:6821/86633","heartbeat_front_addr":"172.21.15.50:6818/86633","state":["exists","up"]},{"osd":4,"uuid":"de6773bd-d286-4ab9-b91f-8a90ee3073d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":86632},{"type":"v1","addr":"172.21.15.50:6806","nonce":86632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":86632},{"type":"v1","addr":"172.21.15.50:6814","nonce":86632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":86632},{"type":"v1","addr":"172.21.15.50:6823","nonce":86632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":86632},{"type":"v1","addr":"172.21.15.50:6820","nonce":86632}]},"public_addr":"172.21.15.50:6806/86632","cluster_addr":"172.21.15.50:6814/86632","heartbeat_back_addr":"172.21.15.50:6823/86632","heartbeat_front_addr":"172.21.15.50:6820/86632","state":["exists","up"]},{"osd":5,"uuid":"e6a847ab-276e-4d54-9a8f-6df5b8ab54cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":86634},{"type":"v1","addr":"172.21.15.50:6801","nonce":86634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":86634},{"type":"v1","addr":"172.21.15.50:6805","nonce":86634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":86634},{"type":"v1","addr":"172.21.15.50:6816","nonce":86634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":86634},{"type":"v1","addr":"172.21.15.50:6811","nonce":86634}]},"public_addr":"172.21.15.50:6801/86634","cluster_addr":"172.21.15.50:6805/86634","heartbeat_back_addr":"172.21.15.50:6816/86634","heartbeat_front_addr":"172.21.15.50:6811/86634","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.819964+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.795270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.734518+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.507975+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.783023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.746894+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:59:54.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.098+0000 7f242768f700 1 -- 172.21.15.7:0/1711397234 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f240c0357e0 msgr2=0x7f240c037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:54.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.098+0000 7f242768f700 1 --2- 172.21.15.7:0/1711397234 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f240c0357e0 0x7f240c037c90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f24180093a0 tx=0x7f2418016000 comp rx=0 tx=0).stop 2023-09-30T01:59:54.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.098+0000 7f242768f700 10 monclient: shutdown 2023-09-30T01:59:54.098 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.098+0000 7f242768f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:54.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.098+0000 7f242768f700 1 -- 172.21.15.7:0/1711397234 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2420096e20 msgr2=0x7f242009cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:54.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.098+0000 7f242768f700 1 --2- 172.21.15.7:0/1711397234 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2420096e20 0x7f242009cbf0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f241400c540 tx=0x7f2414007ea0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.098+0000 7f242768f700 1 -- 172.21.15.7:0/1711397234 shutdown_connections 2023-09-30T01:59:54.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.098+0000 7f242768f700 1 --2- 172.21.15.7:0/1711397234 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f240c0357e0 0x7f240c037c90 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.098+0000 7f242768f700 1 --2- 172.21.15.7:0/1711397234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2420096a50 0x7f24200a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.098+0000 7f242768f700 1 --2- 172.21.15.7:0/1711397234 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2420137710 0x7f242009d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.098+0000 7f242768f700 1 --2- 172.21.15.7:0/1711397234 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f2420096e20 0x7f242009cbf0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.098+0000 7f242768f700 1 -- 172.21.15.7:0/1711397234 >> 172.21.15.7:0/1711397234 conn(0x7f2420070cf0 msgr2=0x7f242012f110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:54.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.098+0000 7f242768f700 1 -- 172.21.15.7:0/1711397234 shutdown_connections 2023-09-30T01:59:54.099 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.098+0000 7f242768f700 1 -- 172.21.15.7:0/1711397234 wait complete. 2023-09-30T01:59:54.110 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T01:59:54.110 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:59:54.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.231+0000 7ff5e1223700 10 monclient: get_monmap_and_config 2023-09-30T01:59:54.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.231+0000 7ff5e1223700 10 monclient: build_initial_monmap 2023-09-30T01:59:54.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.231+0000 7ff5e1223700 10 monclient: monmap: 2023-09-30T01:59:54.231 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:54.231 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:54.231 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:54.232268+0000 2023-09-30T01:59:54.232 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:54.232268+0000 2023-09-30T01:59:54.232 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:54.232 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:54.232 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:54.232 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:54.232 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:54.232 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:54.232 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:54.232 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.231+0000 7ff5e1223700 1 Processor -- start 2023-09-30T01:59:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.231+0000 7ff5e1223700 1 -- start start 2023-09-30T01:59:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.231+0000 7ff5e1223700 10 monclient: init 2023-09-30T01:59:54.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5e1223700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:54.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5e1223700 10 monclient: _add_conns ranks=[4,3,1,0,2] 2023-09-30T01:59:54.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5e1223700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7ff5dc096a50 0x7ff5dc1327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5e1223700 10 monclient(hunting): picked mon.noname-c con 0x7ff5dc096a50 addr v2:172.21.15.50:3300/0 2023-09-30T01:59:54.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5e1223700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff5dc0a2390 0x7ff5dc096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5e1223700 10 monclient(hunting): picked mon.noname-a con 0x7ff5dc0a2390 addr v2:172.21.15.7:3300/0 2023-09-30T01:59:54.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5e1223700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff5dc097360 addr v1:172.21.15.50:6789/0 2023-09-30T01:59:54.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5e1223700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5dc0980e0 con 0x7ff5dc097360 2023-09-30T01:59:54.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5e1223700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:54.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5e1223700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5dc0991a0 con 0x7ff5dc0a2390 2023-09-30T01:59:54.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5e1223700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:54.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5e1223700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5dc099480 con 0x7ff5dc096a50 2023-09-30T01:59:54.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5e1223700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:54.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5e1223700 10 monclient(hunting): authenticate will time out at 1483.297288s 2023-09-30T01:59:54.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5da59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff5dc0a2390 0x7ff5dc096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5da59c700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7ff5dc0a2390 0x7ff5dc096e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49284/0 (socket says 172.21.15.7:49284) 2023-09-30T01:59:54.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5da59c700 1 -- 172.21.15.7:0/1085180820 learned_addr learned my addr 172.21.15.7:0/1085180820 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:54.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5da59c700 10 monclient(hunting): get_auth_request con 0x7ff5dc0a2390 auth_method 0 2023-09-30T01:59:54.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5da59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:54.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5da59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:54.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5da59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:54.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5da59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:54.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5da59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:54.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5da59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:54.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5da59c700 10 monclient(hunting): handle_auth_done global_id 4734 payload 274 2023-09-30T01:59:54.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5dad9d700 1 --2- 172.21.15.7:0/1085180820 >> v2:172.21.15.50:3300/0 conn(0x7ff5dc096a50 0x7ff5dc1327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5da59c700 1 -- 172.21.15.7:0/1085180820 >> v2:172.21.15.50:3300/0 conn(0x7ff5dc096a50 msgr2=0x7ff5dc1327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:54.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5da59c700 1 --2- 172.21.15.7:0/1085180820 >> v2:172.21.15.50:3300/0 conn(0x7ff5dc096a50 0x7ff5dc1327a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5da59c700 1 -- 172.21.15.7:0/1085180820 >> v1:172.21.15.50:6789/0 conn(0x7ff5dc097360 legacy=0x7ff5dc12ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5da59c700 10 monclient: _finish_hunting 0 2023-09-30T01:59:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5da59c700 1 monclient: found mon.noname-a 2023-09-30T01:59:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5da59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5da59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5da59c700 1 -- 172.21.15.7:0/1085180820 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5dc097bf0 con 0x7ff5dc0a2390 2023-09-30T01:59:54.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5dad9d700 10 monclient: get_auth_request con 0x7ff5dc096a50 auth_method 0 2023-09-30T01:59:54.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.232+0000 7ff5dad9d700 1 --2- 172.21.15.7:0/1085180820 >> v2:172.21.15.50:3300/0 conn(0x7ff5dc096a50 0x7ff5dc1327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:59:54.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.234+0000 7ff5da59c700 1 -- 172.21.15.7:0/1085180820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc0a2390 msgr2=0x7ff5dc096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:54.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.234+0000 7ff5da59c700 1 --2- 172.21.15.7:0/1085180820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc0a2390 0x7ff5dc096e20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc0a2b00 tx=0x7ff5d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81fac58c65960f56 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:54.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5d9d9b700 1 -- 172.21.15.7:0/1085180820 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d000ba30 con 0x7ff5dc0a2390 2023-09-30T01:59:54.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5d9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:54.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5d9d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:54.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5d9d9b700 10 monclient: dump: 2023-09-30T01:59:54.241 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:54.242 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:54.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5d9d9b700 10 monclient: _finish_auth 0 2023-09-30T01:59:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5d9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5e1223700 5 monclient: authenticate success, global_id 4734 2023-09-30T01:59:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5d9d9b700 1 -- 172.21.15.7:0/1085180820 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d000bb90 con 0x7ff5dc0a2390 2023-09-30T01:59:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5e1223700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5d9d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5e1223700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5d9d9b700 1 -- 172.21.15.7:0/1085180820 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0011420 con 0x7ff5dc0a2390 2023-09-30T01:59:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5e1223700 4 set_mon_vals no callback set 2023-09-30T01:59:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5d9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:54.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5d9d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5d9d9b700 10 monclient: dump: 2023-09-30T01:59:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:54.244 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:54.244 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:54.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5e1223700 10 monclient: shutdown 2023-09-30T01:59:54.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5e1223700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:54.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5e1223700 1 -- 172.21.15.7:0/1085180820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc0a2390 msgr2=0x7ff5dc096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:54.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5e1223700 1 --2- 172.21.15.7:0/1085180820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc0a2390 0x7ff5dc096e20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc0a2b00 tx=0x7ff5d0009250 comp rx=0 tx=0).stop 2023-09-30T01:59:54.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5e1223700 1 -- 172.21.15.7:0/1085180820 shutdown_connections 2023-09-30T01:59:54.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5e1223700 1 --2- 172.21.15.7:0/1085180820 >> v2:172.21.15.50:3300/0 conn(0x7ff5dc096a50 0x7ff5dc1327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5e1223700 1 --2- 172.21.15.7:0/1085180820 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc0a2390 0x7ff5dc096e20 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5e1223700 1 -- 172.21.15.7:0/1085180820 >> 172.21.15.7:0/1085180820 conn(0x7ff5dc070cf0 msgr2=0x7ff5dc0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:54.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5e1223700 1 -- 172.21.15.7:0/1085180820 shutdown_connections 2023-09-30T01:59:54.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.235+0000 7ff5e1223700 1 -- 172.21.15.7:0/1085180820 wait complete. 2023-09-30T01:59:54.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.236+0000 7ff5e1223700 10 monclient: build_initial_monmap 2023-09-30T01:59:54.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.236+0000 7ff5e1223700 10 monclient: monmap: 2023-09-30T01:59:54.246 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:54.246 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:54.246 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:54.246 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:54.246 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:54.246 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:54.246 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:54.246 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:54.247 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:54.247 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:54.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.236+0000 7ff5e1223700 1 Processor -- start 2023-09-30T01:59:54.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.236+0000 7ff5e1223700 1 -- start start 2023-09-30T01:59:54.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.236+0000 7ff5e1223700 10 monclient: init 2023-09-30T01:59:54.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5e1223700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:54.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5e1223700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:59:54.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5e1223700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5dc096a50 0x7ff5dc0a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5e1223700 10 monclient(hunting): picked mon.noname-c con 0x7ff5dc096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:54.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5e1223700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc097360 0x7ff5dc11c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5e1223700 10 monclient(hunting): picked mon.noname-a con 0x7ff5dc097360 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:54.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5e1223700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5dc0a2390 0x7ff5dc12e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5e1223700 10 monclient(hunting): picked mon.noname-b con 0x7ff5dc0a2390 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:54.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5e1223700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:54.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5e1223700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc11c770 con 0x7ff5dc097360 2023-09-30T01:59:54.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5e1223700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:54.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5e1223700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0a1a80 con 0x7ff5dc0a2390 2023-09-30T01:59:54.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5e1223700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:54.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5e1223700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc11cda0 con 0x7ff5dc096a50 2023-09-30T01:59:54.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5e1223700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:54.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5e1223700 10 monclient(hunting): authenticate will time out at 1483.302229s 2023-09-30T01:59:54.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5dad9d700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5dc096a50 0x7ff5dc0a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5dad9d700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5dc096a50 0x7ff5dc0a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41252/0 (socket says 172.21.15.7:41252) 2023-09-30T01:59:54.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5dad9d700 1 -- 172.21.15.7:0/1245362575 learned_addr learned my addr 172.21.15.7:0/1245362575 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:54.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5da59c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc097360 0x7ff5dc11c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5dad9d700 10 monclient(hunting): get_auth_request con 0x7ff5dc096a50 auth_method 0 2023-09-30T01:59:54.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5dad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:54.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5dad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:54.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5dad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:54.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5da59c700 10 monclient(hunting): get_auth_request con 0x7ff5dc097360 auth_method 0 2023-09-30T01:59:54.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5da59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:54.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5da59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:54.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5da59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:54.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5dad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:54.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5dad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:54.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5dad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:54.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5da59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:54.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5da59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:54.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5da59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:54.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5db59e700 1 --2- 172.21.15.7:0/1245362575 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5dc0a2390 0x7ff5dc12e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5db59e700 10 monclient(hunting): get_auth_request con 0x7ff5dc0a2390 auth_method 0 2023-09-30T01:59:54.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5db59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:54.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5db59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:54.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5db59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:54.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5dad9d700 10 monclient(hunting): handle_auth_done global_id 4757 payload 824 2023-09-30T01:59:54.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5dad9d700 1 -- 172.21.15.7:0/1245362575 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5dc0a2390 msgr2=0x7ff5dc12e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:54.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5dad9d700 1 --2- 172.21.15.7:0/1245362575 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5dc0a2390 0x7ff5dc12e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5dad9d700 1 -- 172.21.15.7:0/1245362575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc097360 msgr2=0x7ff5dc11c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:54.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5dad9d700 1 --2- 172.21.15.7:0/1245362575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc097360 0x7ff5dc11c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5dad9d700 10 monclient: _finish_hunting 0 2023-09-30T01:59:54.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5dad9d700 1 monclient: found mon.noname-c 2023-09-30T01:59:54.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5dad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:54.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5dad9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:59:54.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.237+0000 7ff5dad9d700 1 -- 172.21.15.7:0/1245362575 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d00094f0 con 0x7ff5dc096a50 2023-09-30T01:59:54.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.239+0000 7ff5dad9d700 1 -- 172.21.15.7:0/1245362575 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5dc096a50 msgr2=0x7ff5dc0a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:54.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.239+0000 7ff5dad9d700 1 --2- 172.21.15.7:0/1245362575 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5dc096a50 0x7ff5dc0a0dd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc00c830 tx=0x7ff5cc008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:54.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.239+0000 7ff5c2ffd700 1 -- 172.21.15.7:0/1245362575 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc011810 con 0x7ff5dc096a50 2023-09-30T01:59:54.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.239+0000 7ff5c2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:54.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.240+0000 7ff5c2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:59:54.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.240+0000 7ff5c2ffd700 10 monclient: dump: 2023-09-30T01:59:54.259 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:54.259 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:54.259 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.259 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.259 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:54.259 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:54.259 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:54.259 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:54.259 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:54.259 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:54.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.240+0000 7ff5c2ffd700 10 monclient: _finish_auth 0 2023-09-30T01:59:54.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.240+0000 7ff5c2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:54.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.240+0000 7ff5e1223700 5 monclient: authenticate success, global_id 4757 2023-09-30T01:59:54.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.240+0000 7ff5c2ffd700 1 -- 172.21.15.7:0/1245362575 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5cc011970 con 0x7ff5dc096a50 2023-09-30T01:59:54.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.240+0000 7ff5c2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:54.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.240+0000 7ff5e1223700 10 monclient: _renew_subs 2023-09-30T01:59:54.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.240+0000 7ff5c2ffd700 1 -- 172.21.15.7:0/1245362575 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc011d30 con 0x7ff5dc096a50 2023-09-30T01:59:54.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.240+0000 7ff5e1223700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:54.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.240+0000 7ff5e1223700 1 -- 172.21.15.7:0/1245362575 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5dc11e390 con 0x7ff5dc096a50 2023-09-30T01:59:54.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.240+0000 7ff5d959a700 4 set_mon_vals no callback set 2023-09-30T01:59:54.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.240+0000 7ff5c2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:54.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.240+0000 7ff5c2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:54.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.240+0000 7ff5c2ffd700 10 monclient: dump: 2023-09-30T01:59:54.260 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:54.261 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:54.261 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.261 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.261 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:54.261 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:54.261 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:54.261 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:54.261 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:54.261 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:54.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.240+0000 7ff5e1223700 10 monclient: _renew_subs 2023-09-30T01:59:54.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.240+0000 7ff5e1223700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:54.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.240+0000 7ff5e1223700 1 -- 172.21.15.7:0/1245362575 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5dc12e780 con 0x7ff5dc096a50 2023-09-30T01:59:54.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.241+0000 7ff5e1223700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:54.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.241+0000 7ff5e1223700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:54.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.241+0000 7ff5e1223700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:54.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.241+0000 7ff5c2ffd700 1 -- 172.21.15.7:0/1245362575 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7ff5cc0195c0 con 0x7ff5dc096a50 2023-09-30T01:59:54.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.241+0000 7ff5e1223700 1 -- 172.21.15.7:0/1245362575 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5dc064c10 con 0x7ff5dc096a50 2023-09-30T01:59:54.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.241+0000 7ff5c2ffd700 1 --2- 172.21.15.7:0/1245362575 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff5c80357e0 0x7ff5c8037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.241+0000 7ff5c2ffd700 1 -- 172.21.15.7:0/1245362575 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7ff5cc047d50 con 0x7ff5dc096a50 2023-09-30T01:59:54.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.241+0000 7ff5da59c700 1 --2- 172.21.15.7:0/1245362575 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff5c80357e0 0x7ff5c8037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.241+0000 7ff5da59c700 10 monclient: get_auth_request con 0x7ff5c80357e0 auth_method 0 2023-09-30T01:59:54.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.244+0000 7ff5da59c700 1 --2- 172.21.15.7:0/1245362575 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff5c80357e0 0x7ff5c8037c90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc0a2b00 tx=0x7ff5d0016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:54.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.245+0000 7ff5c2ffd700 1 -- 172.21.15.7:0/1245362575 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff5cc021710 con 0x7ff5dc096a50 2023-09-30T01:59:54.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.245+0000 7ff5c2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:54.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.245+0000 7ff5c2ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:54.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.433+0000 7ff5e1223700 1 -- 172.21.15.7:0/1245362575 --> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff5dc11d5a0 con 0x7ff5c80357e0 2023-09-30T01:59:54.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.435+0000 7ff5c2ffd700 1 -- 172.21.15.7:0/1245362575 <== mgr.4105 v2:172.21.15.50:6824/86200 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53791 (secure 0 0 0) 0x7ff5dc11d5a0 con 0x7ff5c80357e0 2023-09-30T01:59:54.436 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:59:54.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.437+0000 7ff5e1223700 1 -- 172.21.15.7:0/1245362575 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff5c80357e0 msgr2=0x7ff5c8037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:54.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.437+0000 7ff5e1223700 1 --2- 172.21.15.7:0/1245362575 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff5c80357e0 0x7ff5c8037c90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc0a2b00 tx=0x7ff5d0016000 comp rx=0 tx=0).stop 2023-09-30T01:59:54.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.437+0000 7ff5e1223700 10 monclient: shutdown 2023-09-30T01:59:54.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.437+0000 7ff5e1223700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:54.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.437+0000 7ff5e1223700 1 -- 172.21.15.7:0/1245362575 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5dc096a50 msgr2=0x7ff5dc0a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:54.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.437+0000 7ff5e1223700 1 --2- 172.21.15.7:0/1245362575 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5dc096a50 0x7ff5dc0a0dd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc00c830 tx=0x7ff5cc008040 comp rx=0 tx=0).stop 2023-09-30T01:59:54.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.437+0000 7ff5e1223700 1 -- 172.21.15.7:0/1245362575 shutdown_connections 2023-09-30T01:59:54.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.437+0000 7ff5e1223700 1 --2- 172.21.15.7:0/1245362575 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7ff5c80357e0 0x7ff5c8037c90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.437+0000 7ff5e1223700 1 --2- 172.21.15.7:0/1245362575 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5dc0a2390 0x7ff5dc12e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.437+0000 7ff5e1223700 1 --2- 172.21.15.7:0/1245362575 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7ff5dc096a50 0x7ff5dc0a0dd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.437+0000 7ff5e1223700 1 --2- 172.21.15.7:0/1245362575 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc097360 0x7ff5dc11c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.437+0000 7ff5e1223700 1 -- 172.21.15.7:0/1245362575 >> 172.21.15.7:0/1245362575 conn(0x7ff5dc070cf0 msgr2=0x7ff5dc131640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:54.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.438+0000 7ff5e1223700 1 -- 172.21.15.7:0/1245362575 shutdown_connections 2023-09-30T01:59:54.439 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.438+0000 7ff5e1223700 1 -- 172.21.15.7:0/1245362575 wait complete. 2023-09-30T01:59:54.439 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-09-30T01:59:54.449 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2023-09-30T01:59:54.328777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":32644,"kb_used_data":1876,"kb_used_omap":0,"kb_used_meta":30720,"kb_avail":566198396,"statfs":{"total":579820584960,"available":579787157504,"internally_reserved":0,"allocated":1921024,"data_stored":1316285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31457280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002591"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.042899+0000","last_change":"2023-09-30T01:54:47.042422+0000","last_active":"2023-09-30T01:54:47.042899+0000","last_peered":"2023-09-30T01:54:47.042899+0000","last_clean":"2023-09-30T01:54:47.042899+0000","last_became_active":"2023-09-30T01:54:47.042183+0000","last_became_peered":"2023-09-30T01:54:47.042183+0000","last_unstale":"2023-09-30T01:54:47.042899+0000","last_undegraded":"2023-09-30T01:54:47.042899+0000","last_fullsized":"2023-09-30T01:54:47.042899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:14:57.725898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.897849+0000","last_change":"2023-09-30T01:54:46.896909+0000","last_active":"2023-09-30T01:54:46.897849+0000","last_peered":"2023-09-30T01:54:46.897849+0000","last_clean":"2023-09-30T01:54:46.897849+0000","last_became_active":"2023-09-30T01:54:46.896620+0000","last_became_peered":"2023-09-30T01:54:46.896620+0000","last_unstale":"2023-09-30T01:54:46.897849+0000","last_undegraded":"2023-09-30T01:54:46.897849+0000","last_fullsized":"2023-09-30T01:54:46.897849+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:28:25.053882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.042370+0000","last_change":"2023-09-30T01:54:47.041973+0000","last_active":"2023-09-30T01:54:47.042370+0000","last_peered":"2023-09-30T01:54:47.042370+0000","last_clean":"2023-09-30T01:54:47.042370+0000","last_became_active":"2023-09-30T01:54:47.041671+0000","last_became_peered":"2023-09-30T01:54:47.041671+0000","last_unstale":"2023-09-30T01:54:47.042370+0000","last_undegraded":"2023-09-30T01:54:47.042370+0000","last_fullsized":"2023-09-30T01:54:47.042370+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:08:29.771169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.896726+0000","last_change":"2023-09-30T01:54:46.896432+0000","last_active":"2023-09-30T01:54:46.896726+0000","last_peered":"2023-09-30T01:54:46.896726+0000","last_clean":"2023-09-30T01:54:46.896726+0000","last_became_active":"2023-09-30T01:54:46.896191+0000","last_became_peered":"2023-09-30T01:54:46.896191+0000","last_unstale":"2023-09-30T01:54:46.896726+0000","last_undegraded":"2023-09-30T01:54:46.896726+0000","last_fullsized":"2023-09-30T01:54:46.896726+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:36:51.492282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":160,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.202781+0000","last_change":"2023-09-30T01:54:45.896629+0000","last_active":"2023-09-30T01:54:47.202781+0000","last_peered":"2023-09-30T01:54:47.202781+0000","last_clean":"2023-09-30T01:54:47.202781+0000","last_became_active":"2023-09-30T01:54:45.896315+0000","last_became_peered":"2023-09-30T01:54:45.896315+0000","last_unstale":"2023-09-30T01:54:47.202781+0000","last_undegraded":"2023-09-30T01:54:47.202781+0000","last_fullsized":"2023-09-30T01:54:47.202781+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:44.377605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:44.377605+0000","last_clean_scrub_stamp":"2023-09-30T01:54:44.377605+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T13:48:02.317525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.899133+0000","last_change":"2023-09-30T01:54:46.898655+0000","last_active":"2023-09-30T01:54:46.899133+0000","last_peered":"2023-09-30T01:54:46.899133+0000","last_clean":"2023-09-30T01:54:46.899133+0000","last_became_active":"2023-09-30T01:54:46.898402+0000","last_became_peered":"2023-09-30T01:54:46.898402+0000","last_unstale":"2023-09-30T01:54:46.899133+0000","last_undegraded":"2023-09-30T01:54:46.899133+0000","last_fullsized":"2023-09-30T01:54:46.899133+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:07:41.886018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.897962+0000","last_change":"2023-09-30T01:54:46.896375+0000","last_active":"2023-09-30T01:54:46.897962+0000","last_peered":"2023-09-30T01:54:46.897962+0000","last_clean":"2023-09-30T01:54:46.897962+0000","last_became_active":"2023-09-30T01:54:46.895973+0000","last_became_peered":"2023-09-30T01:54:46.895973+0000","last_unstale":"2023-09-30T01:54:46.897962+0000","last_undegraded":"2023-09-30T01:54:46.897962+0000","last_fullsized":"2023-09-30T01:54:46.897962+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:52:22.627216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.898698+0000","last_change":"2023-09-30T01:54:46.898322+0000","last_active":"2023-09-30T01:54:46.898698+0000","last_peered":"2023-09-30T01:54:46.898698+0000","last_clean":"2023-09-30T01:54:46.898698+0000","last_became_active":"2023-09-30T01:54:46.898101+0000","last_became_peered":"2023-09-30T01:54:46.898101+0000","last_unstale":"2023-09-30T01:54:46.898698+0000","last_undegraded":"2023-09-30T01:54:46.898698+0000","last_fullsized":"2023-09-30T01:54:46.898698+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:45:31.230458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.042677+0000","last_change":"2023-09-30T01:54:47.042174+0000","last_active":"2023-09-30T01:54:47.042677+0000","last_peered":"2023-09-30T01:54:47.042677+0000","last_clean":"2023-09-30T01:54:47.042677+0000","last_became_active":"2023-09-30T01:54:47.041914+0000","last_became_peered":"2023-09-30T01:54:47.041914+0000","last_unstale":"2023-09-30T01:54:47.042677+0000","last_undegraded":"2023-09-30T01:54:47.042677+0000","last_fullsized":"2023-09-30T01:54:47.042677+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:16:46.356909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5076,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366764,"statfs":{"total":96636764160,"available":96631566336,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52500000000000002,"15min":0.54300000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.69199999999999995,"5min":0.747,"15min":0.747},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53500000000000003,"15min":0.56100000000000005},"min":{"1min":0.40799999999999997,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66400000000000003,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.65100000000000002}]},{"osd":1,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.51500000000000001,"15min":0.53700000000000003},"min":{"1min":0.376,"5min":0.313,"15min":0.313},"max":{"1min":0.66400000000000003,"5min":0.751,"15min":0.751},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.53500000000000003,"15min":0.56100000000000005},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.629,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.55300000000000005}]},{"osd":2,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55100000000000005,"15min":0.57299999999999995},"min":{"1min":0.42499999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.73899999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55300000000000005,"15min":0.56499999999999995},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72599999999999998,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.57899999999999996}]},{"osd":3,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41999999999999998,"15min":0.443},"min":{"1min":0.29899999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.60499999999999998,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45500000000000002,"15min":0.47899999999999998},"min":{"1min":0.312,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.439}]},{"osd":4,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.45500000000000002,"15min":0.48999999999999999},"min":{"1min":0.311,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.50900000000000001,"15min":0.53500000000000003},"min":{"1min":0.36599999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.76700000000000002,"5min":0.874,"15min":0.874},"last":0.46000000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640319,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5076,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366764,"statfs":{"total":96636764160,"available":96631566336,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:58:51 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.30599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.63800000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.50700000000000001,"15min":0.48499999999999999},"min":{"1min":0.37,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66000000000000003,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.47799999999999998}]},{"osd":1,"last update":"Sat Sep 30 01:58:51 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51000000000000001,"15min":0.48799999999999999},"min":{"1min":0.41999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68200000000000005,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.52500000000000002},"min":{"1min":0.47499999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.623,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.52200000000000002}]},{"osd":2,"last update":"Sat Sep 30 01:58:51 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.52300000000000002,"15min":0.504},"min":{"1min":0.38600000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.64600000000000002,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54700000000000004,"15min":0.52500000000000002},"min":{"1min":0.45700000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.69999999999999996,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.53400000000000003}]},{"osd":3,"last update":"Sat Sep 30 01:58:51 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.309,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.54400000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45300000000000001,"15min":0.45700000000000002},"min":{"1min":0.35599999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.374}]},{"osd":5,"last update":"Sat Sep 30 01:58:51 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45000000000000001,"15min":0.45400000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.65300000000000002,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.498,"15min":0.505},"min":{"1min":0.32700000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66200000000000003,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.35799999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640319,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6108,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365732,"statfs":{"total":96636764160,"available":96630509568,"internally_reserved":0,"allocated":675840,"data_stored":572143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.36799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.70199999999999996,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.53800000000000003,"15min":0.54100000000000004},"min":{"1min":0.35899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.71499999999999997,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.746}]},{"osd":1,"last update":"Sat Sep 30 01:58:53 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51400000000000001,"15min":0.51200000000000001},"min":{"1min":0.36599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.622},{"interface":"front","average":{"1min":0.503,"5min":0.50600000000000001,"15min":0.497},"min":{"1min":0.38400000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.63500000000000001,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.70999999999999996}]},{"osd":2,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.51500000000000001,"15min":0.51100000000000001},"min":{"1min":0.434,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.64100000000000001,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54000000000000004,"15min":0.54800000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69299999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.66800000000000004}]},{"osd":4,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.26900000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.59499999999999997,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.48899999999999999,"15min":0.501},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.60599999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.41499999999999998}]},{"osd":5,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.45800000000000002,"15min":0.47399999999999998},"min":{"1min":0.28599999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65400000000000003,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.441,"5min":0.46999999999999997,"15min":0.48199999999999998},"min":{"1min":0.31900000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.59999999999999998,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.54900000000000004}]}]},{"osd":2,"up_from":11,"seq":47244640320,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5140,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366700,"statfs":{"total":96636764160,"available":96631500800,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35299999999999998},"min":{"1min":0.245,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.47999999999999998,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.39300000000000002,"15min":0.38500000000000001},"min":{"1min":0.28100000000000003,"5min":0.254,"15min":0.254},"max":{"1min":0.54300000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.318}]},{"osd":1,"last update":"Sat Sep 30 01:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36799999999999999,"15min":0.37},"min":{"1min":0.23999999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.55800000000000005,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.374,"15min":0.36199999999999999},"min":{"1min":0.251,"5min":0.222,"15min":0.222},"max":{"1min":0.56799999999999995,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.29999999999999999}]},{"osd":3,"last update":"Sat Sep 30 01:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.43099999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.56599999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.54400000000000004}]},{"osd":4,"last update":"Sat Sep 30 01:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.436,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.82699999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.443},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59599999999999997,"15min":0.58399999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.52100000000000002}]},{"osd":5,"last update":"Sat Sep 30 01:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57999999999999996,"15min":0.56999999999999995},"min":{"1min":0.44700000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61499999999999999,"15min":0.61299999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.93799999999999994,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.53400000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640319,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6104,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365736,"statfs":{"total":96636764160,"available":96630513664,"internally_reserved":0,"allocated":671744,"data_stored":571282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34000000000000002,"15min":0.33400000000000002},"min":{"1min":0.219,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.54300000000000004,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.224},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.39500000000000002,"15min":0.39700000000000002},"min":{"1min":0.26700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.59899999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.30299999999999999}]},{"osd":2,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.253,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.51600000000000001,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42199999999999999,"15min":0.42799999999999999},"min":{"1min":0.29899999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.56499999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.34699999999999998}]},{"osd":3,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.56799999999999995,"15min":0.56399999999999995},"min":{"1min":0.35199999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.73999999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.73299999999999998,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.61599999999999999}]},{"osd":4,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.57599999999999996,"15min":0.57899999999999996},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.78900000000000003,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57499999999999996,"15min":0.57799999999999996},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.77900000000000003,"5min":1.026,"15min":1.026},"last":0.51400000000000001}]},{"osd":5,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57199999999999995,"15min":0.57799999999999996},"min":{"1min":0.40999999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76100000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.746,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.66500000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640321,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5140,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366700,"statfs":{"total":96636764160,"available":96631500800,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:58:52 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.32800000000000001,"15min":0.308},"min":{"1min":0.23799999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.49299999999999999,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.33300000000000002,"15min":0.317},"min":{"1min":0.248,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54600000000000004,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.50900000000000001}]},{"osd":2,"last update":"Sat Sep 30 01:58:52 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.35199999999999998,"15min":0.33500000000000002},"min":{"1min":0.28599999999999998,"5min":0.22,"15min":0.22},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.371,"15min":0.36299999999999999},"min":{"1min":0.26800000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.57099999999999995,"5min":0.62,"15min":0.62},"last":0.48699999999999999}]},{"osd":3,"last update":"Sat Sep 30 01:58:52 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.52900000000000003,"15min":0.51500000000000001},"min":{"1min":0.40799999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71099999999999997,"5min":0.748,"15min":0.748},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53700000000000003,"15min":0.53200000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82899999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.60099999999999998}]},{"osd":4,"last update":"Sat Sep 30 01:58:52 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56899999999999995,"15min":0.56200000000000006},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.44700000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.77700000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.63400000000000001}]},{"osd":5,"last update":"Sat Sep 30 01:58:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56100000000000005,"15min":0.55200000000000005},"min":{"1min":0.39600000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.77100000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.58099999999999996,"15min":0.56799999999999995},"min":{"1min":0.42299999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.78400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:59:54.451 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:59:54.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.568+0000 7f50b8e9d700 10 monclient: get_monmap_and_config 2023-09-30T01:59:54.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.568+0000 7f50b8e9d700 10 monclient: build_initial_monmap 2023-09-30T01:59:54.576 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.568+0000 7f50b8e9d700 10 monclient: monmap: 2023-09-30T01:59:54.576 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:54.576 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:54.577 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:54.569689+0000 2023-09-30T01:59:54.577 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:54.569689+0000 2023-09-30T01:59:54.577 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:54.577 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:54.577 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:54.577 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:54.577 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:54.577 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:54.577 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:54.577 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:54.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.569+0000 7f50b8e9d700 1 Processor -- start 2023-09-30T01:59:54.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.569+0000 7f50b8e9d700 1 -- start start 2023-09-30T01:59:54.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.569+0000 7f50b8e9d700 10 monclient: init 2023-09-30T01:59:54.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.569+0000 7f50b8e9d700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:54.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.569+0000 7f50b8e9d700 10 monclient: _add_conns ranks=[4,1,2,0,3] 2023-09-30T01:59:54.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.569+0000 7f50b8e9d700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f50b4098540 0x7f50b41329e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.569+0000 7f50b8e9d700 10 monclient(hunting): picked mon.noname-c con 0x7f50b4098540 addr v2:172.21.15.50:3300/0 2023-09-30T01:59:54.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.569+0000 7f50b8e9d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f50b4098910 addr v1:172.21.15.50:6789/0 2023-09-30T01:59:54.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.569+0000 7f50b8e9d700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50b4098d10 0x7f50b412f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.569+0000 7f50b8e9d700 10 monclient(hunting): picked mon.noname-b con 0x7f50b4098d10 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:54.578 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.569+0000 7f50b8e9d700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b4069f10 con 0x7f50b4098910 2023-09-30T01:59:54.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.569+0000 7f50b8e9d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:54.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.569+0000 7f50b8e9d700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b40a0f40 con 0x7f50b4098540 2023-09-30T01:59:54.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.569+0000 7f50b8e9d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:54.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.569+0000 7f50b8e9d700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b4132f20 con 0x7f50b4098d10 2023-09-30T01:59:54.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.569+0000 7f50b8e9d700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:54.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.569+0000 7f50b8e9d700 10 monclient(hunting): authenticate will time out at 1483.634892s 2023-09-30T01:59:54.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b3fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50b4098d10 0x7f50b412f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b3fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50b4098d10 0x7f50b412f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41266/0 (socket says 172.21.15.7:41266) 2023-09-30T01:59:54.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b3fff700 1 -- 172.21.15.7:0/3742716581 learned_addr learned my addr 172.21.15.7:0/3742716581 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:54.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b3fff700 10 monclient(hunting): get_auth_request con 0x7f50b4098d10 auth_method 0 2023-09-30T01:59:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b37fe700 1 --2- 172.21.15.7:0/3742716581 >> v2:172.21.15.50:3300/0 conn(0x7f50b4098540 0x7f50b41329e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b37fe700 10 monclient(hunting): get_auth_request con 0x7f50b4098540 auth_method 0 2023-09-30T01:59:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:54.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:54.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:54.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:54.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:54.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b3fff700 10 monclient(hunting): handle_auth_done global_id 4760 payload 274 2023-09-30T01:59:54.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b3fff700 1 -- 172.21.15.7:0/3742716581 >> v2:172.21.15.50:3300/0 conn(0x7f50b4098540 msgr2=0x7f50b41329e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:54.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b3fff700 1 --2- 172.21.15.7:0/3742716581 >> v2:172.21.15.50:3300/0 conn(0x7f50b4098540 0x7f50b41329e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b3fff700 1 -- 172.21.15.7:0/3742716581 >> v1:172.21.15.50:6789/0 conn(0x7f50b4098910 legacy=0x7f50b412ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:54.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b3fff700 10 monclient: _finish_hunting 0 2023-09-30T01:59:54.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b3fff700 1 monclient: found mon.noname-b 2023-09-30T01:59:54.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:54.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:54.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.570+0000 7f50b3fff700 1 -- 172.21.15.7:0/3742716581 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b4096a50 con 0x7f50b4098d10 2023-09-30T01:59:54.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.572+0000 7f50b3fff700 1 -- 172.21.15.7:0/3742716581 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50b4098d10 msgr2=0x7f50b412f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:54.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.572+0000 7f50b3fff700 1 --2- 172.21.15.7:0/3742716581 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50b4098d10 0x7f50b412f2d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f50ac009f20 tx=0x7f50ac009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91a8c2c17673b163 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:54.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.572+0000 7f50b27fc700 1 -- 172.21.15.7:0/3742716581 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ac00e070 con 0x7f50b4098d10 2023-09-30T01:59:54.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.572+0000 7f50b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:54.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.572+0000 7f50b27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:54.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.572+0000 7f50b27fc700 10 monclient: dump: 2023-09-30T01:59:54.582 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:54.582 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:54.583 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.583 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.583 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:54.583 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:54.583 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:54.583 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:54.583 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:54.583 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:54.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.572+0000 7f50b27fc700 10 monclient: _finish_auth 0 2023-09-30T01:59:54.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.572+0000 7f50b27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:54.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.572+0000 7f50b8e9d700 5 monclient: authenticate success, global_id 4760 2023-09-30T01:59:54.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.572+0000 7f50b8e9d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:54.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.572+0000 7f50b27fc700 1 -- 172.21.15.7:0/3742716581 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50ac00bba0 con 0x7f50b4098d10 2023-09-30T01:59:54.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.572+0000 7f50b27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:54.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.572+0000 7f50b8e9d700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:54.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.572+0000 7f50b8e9d700 4 set_mon_vals no callback set 2023-09-30T01:59:54.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.572+0000 7f50b27fc700 1 -- 172.21.15.7:0/3742716581 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ac0092e0 con 0x7f50b4098d10 2023-09-30T01:59:54.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.572+0000 7f50b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:54.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.572+0000 7f50b27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:54.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.572+0000 7f50b27fc700 10 monclient: dump: 2023-09-30T01:59:54.584 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:54.585 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:54.585 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.585 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.585 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:54.585 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:54.585 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:54.585 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:54.585 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:54.585 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:54.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.573+0000 7f50b8e9d700 10 monclient: shutdown 2023-09-30T01:59:54.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.573+0000 7f50b8e9d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:54.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.573+0000 7f50b8e9d700 1 -- 172.21.15.7:0/3742716581 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50b4098d10 msgr2=0x7f50b412f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:54.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.573+0000 7f50b8e9d700 1 --2- 172.21.15.7:0/3742716581 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50b4098d10 0x7f50b412f2d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f50ac009f20 tx=0x7f50ac009840 comp rx=0 tx=0).stop 2023-09-30T01:59:54.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.573+0000 7f50b8e9d700 1 -- 172.21.15.7:0/3742716581 shutdown_connections 2023-09-30T01:59:54.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.573+0000 7f50b8e9d700 1 --2- 172.21.15.7:0/3742716581 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50b4098d10 0x7f50b412f2d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.573+0000 7f50b8e9d700 1 --2- 172.21.15.7:0/3742716581 >> v2:172.21.15.50:3300/0 conn(0x7f50b4098540 0x7f50b41329e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.573+0000 7f50b8e9d700 1 -- 172.21.15.7:0/3742716581 >> 172.21.15.7:0/3742716581 conn(0x7f50b4071830 msgr2=0x7f50b40a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:54.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.573+0000 7f50b8e9d700 1 -- 172.21.15.7:0/3742716581 shutdown_connections 2023-09-30T01:59:54.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.573+0000 7f50b8e9d700 1 -- 172.21.15.7:0/3742716581 wait complete. 2023-09-30T01:59:54.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.573+0000 7f50b8e9d700 10 monclient: build_initial_monmap 2023-09-30T01:59:54.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.573+0000 7f50b8e9d700 10 monclient: monmap: 2023-09-30T01:59:54.587 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:54.587 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:54.587 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:54.587 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:54.587 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:54.587 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:54.587 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:54.587 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:54.587 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:54.587 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:54.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b8e9d700 1 Processor -- start 2023-09-30T01:59:54.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b8e9d700 1 -- start start 2023-09-30T01:59:54.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b8e9d700 10 monclient: init 2023-09-30T01:59:54.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b8e9d700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:54.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b8e9d700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:59:54.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b8e9d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50b4098540 0x7f50b41415b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b8e9d700 10 monclient(hunting): picked mon.noname-a con 0x7f50b4098540 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:54.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b8e9d700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50b4098910 0x7f50b412f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b8e9d700 10 monclient(hunting): picked mon.noname-c con 0x7f50b4098910 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:54.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b8e9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50b4098d10 0x7f50b412f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b8e9d700 10 monclient(hunting): picked mon.noname-b con 0x7f50b4098d10 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:54.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b8e9d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:54.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b8e9d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b41327b0 con 0x7f50b4098540 2023-09-30T01:59:54.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b8e9d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:54.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b8e9d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b40a3d10 con 0x7f50b4098d10 2023-09-30T01:59:54.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b8e9d700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:54.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b8e9d700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b411f510 con 0x7f50b4098910 2023-09-30T01:59:54.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b8e9d700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:54.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b8e9d700 10 monclient(hunting): authenticate will time out at 1483.639837s 2023-09-30T01:59:54.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50b4098540 0x7f50b41415b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b2ffd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50b4098910 0x7f50b412f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b37fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50b4098540 0x7f50b41415b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49300/0 (socket says 172.21.15.7:49300) 2023-09-30T01:59:54.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b37fe700 1 -- 172.21.15.7:0/1631098947 learned_addr learned my addr 172.21.15.7:0/1631098947 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:54.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b37fe700 10 monclient(hunting): get_auth_request con 0x7f50b4098540 auth_method 0 2023-09-30T01:59:54.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:54.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:54.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b2ffd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50b4098910 0x7f50b412f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41274/0 (socket says 172.21.15.7:41274) 2023-09-30T01:59:54.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:54.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b2ffd700 10 monclient(hunting): get_auth_request con 0x7f50b4098910 auth_method 0 2023-09-30T01:59:54.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.574+0000 7f50b2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:54.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:54.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:54.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:54.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:54.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:54.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b3fff700 1 --2- 172.21.15.7:0/1631098947 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50b4098d10 0x7f50b412f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:54.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:54.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:54.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b3fff700 10 monclient(hunting): get_auth_request con 0x7f50b4098d10 auth_method 0 2023-09-30T01:59:54.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:54.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:54.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:54.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b37fe700 10 monclient(hunting): handle_auth_done global_id 4740 payload 824 2023-09-30T01:59:54.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b37fe700 1 -- 172.21.15.7:0/1631098947 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50b4098910 msgr2=0x7f50b412f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:54.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b37fe700 1 --2- 172.21.15.7:0/1631098947 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50b4098910 0x7f50b412f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b37fe700 1 -- 172.21.15.7:0/1631098947 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50b4098d10 msgr2=0x7f50b412f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:54.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b37fe700 1 --2- 172.21.15.7:0/1631098947 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50b4098d10 0x7f50b412f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b37fe700 10 monclient: _finish_hunting 0 2023-09-30T01:59:54.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b37fe700 1 monclient: found mon.noname-a 2023-09-30T01:59:54.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:54.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b37fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:54.592 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b37fe700 1 -- 172.21.15.7:0/1631098947 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ac009ae0 con 0x7f50b4098540 2023-09-30T01:59:54.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b2ffd700 1 --2- 172.21.15.7:0/1631098947 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50b4098910 0x7f50b412f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:54.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.575+0000 7f50b3fff700 1 --2- 172.21.15.7:0/1631098947 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50b4098d10 0x7f50b412f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:59:54.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f50b37fe700 1 -- 172.21.15.7:0/1631098947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50b4098540 msgr2=0x7f50b41415b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:54.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f50b37fe700 1 --2- 172.21.15.7:0/1631098947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50b4098540 0x7f50b41415b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f50a400ec80 tx=0x7f50a40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:54.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f509b7fe700 1 -- 172.21.15.7:0/1631098947 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a400ca60 con 0x7f50b4098540 2023-09-30T01:59:54.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f509b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:54.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f509b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:54.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f509b7fe700 10 monclient: dump: 2023-09-30T01:59:54.596 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:54.596 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:54.596 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.596 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.596 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:54.596 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:54.596 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:54.596 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:54.596 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:54.596 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:54.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f509b7fe700 10 monclient: _finish_auth 0 2023-09-30T01:59:54.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f509b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:54.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f509b7fe700 1 -- 172.21.15.7:0/1631098947 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a400cbc0 con 0x7f50b4098540 2023-09-30T01:59:54.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f50b8e9d700 5 monclient: authenticate success, global_id 4740 2023-09-30T01:59:54.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f50b8e9d700 10 monclient: _renew_subs 2023-09-30T01:59:54.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f50b8e9d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:54.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f50b8e9d700 1 -- 172.21.15.7:0/1631098947 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b40a05c0 con 0x7f50b4098540 2023-09-30T01:59:54.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f509b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:54.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f509b7fe700 1 -- 172.21.15.7:0/1631098947 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a4013470 con 0x7f50b4098540 2023-09-30T01:59:54.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f509b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:54.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f509b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:54.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f509b7fe700 10 monclient: dump: 2023-09-30T01:59:54.598 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:54.598 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:54.598 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.598 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.598 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:54.598 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:54.598 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:54.598 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:54.598 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:54.598 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:54.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f50b8e9d700 10 monclient: _renew_subs 2023-09-30T01:59:54.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f50b1ffb700 4 set_mon_vals no callback set 2023-09-30T01:59:54.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f50b8e9d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:54.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.577+0000 7f50b8e9d700 1 -- 172.21.15.7:0/1631098947 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b411d640 con 0x7f50b4098540 2023-09-30T01:59:54.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.578+0000 7f509b7fe700 1 -- 172.21.15.7:0/1631098947 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f50a40137e0 con 0x7f50b4098540 2023-09-30T01:59:54.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.578+0000 7f50b8e9d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:54.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.578+0000 7f50b8e9d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:54.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.578+0000 7f50b8e9d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:54.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.578+0000 7f50b8e9d700 1 -- 172.21.15.7:0/1631098947 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b411f870 con 0x7f50b4098540 2023-09-30T01:59:54.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.578+0000 7f509b7fe700 1 --2- 172.21.15.7:0/1631098947 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f50a0035810 0x7f50a0037cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.578+0000 7f509b7fe700 1 -- 172.21.15.7:0/1631098947 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f50a4047ff0 con 0x7f50b4098540 2023-09-30T01:59:54.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.579+0000 7f50b2ffd700 1 --2- 172.21.15.7:0/1631098947 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f50a0035810 0x7f50a0037cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.579+0000 7f50b2ffd700 10 monclient: get_auth_request con 0x7f50a0035810 auth_method 0 2023-09-30T01:59:54.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.581+0000 7f50b2ffd700 1 --2- 172.21.15.7:0/1631098947 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f50a0035810 0x7f50a0037cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f50b411d950 tx=0x7f50a8007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:54.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.583+0000 7f509b7fe700 1 -- 172.21.15.7:0/1631098947 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f50b411f870 con 0x7f50b4098540 2023-09-30T01:59:54.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.583+0000 7f509b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:54.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.583+0000 7f509b7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:54.770 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.770+0000 7f50b8e9d700 1 -- 172.21.15.7:0/1631098947 --> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f50b411f870 con 0x7f50a0035810 2023-09-30T01:59:54.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.773+0000 7f509b7fe700 1 -- 172.21.15.7:0/1631098947 <== mgr.4105 v2:172.21.15.50:6824/86200 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53791 (secure 0 0 0) 0x7f50b411f870 con 0x7f50a0035810 2023-09-30T01:59:54.773 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:59:54.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.774+0000 7f50b8e9d700 1 -- 172.21.15.7:0/1631098947 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f50a0035810 msgr2=0x7f50a0037cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:54.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.774+0000 7f50b8e9d700 1 --2- 172.21.15.7:0/1631098947 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f50a0035810 0x7f50a0037cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f50b411d950 tx=0x7f50a8007ea0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.774+0000 7f50b8e9d700 10 monclient: shutdown 2023-09-30T01:59:54.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.774+0000 7f50b8e9d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:54.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.774+0000 7f50b8e9d700 1 -- 172.21.15.7:0/1631098947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50b4098540 msgr2=0x7f50b41415b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:54.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.774+0000 7f50b8e9d700 1 --2- 172.21.15.7:0/1631098947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50b4098540 0x7f50b41415b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f50a400ec80 tx=0x7f50a40091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.774+0000 7f50b8e9d700 1 -- 172.21.15.7:0/1631098947 shutdown_connections 2023-09-30T01:59:54.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.774+0000 7f50b8e9d700 1 --2- 172.21.15.7:0/1631098947 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f50a0035810 0x7f50a0037cc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.774+0000 7f50b8e9d700 1 --2- 172.21.15.7:0/1631098947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50b4098540 0x7f50b41415b0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.774+0000 7f50b8e9d700 1 --2- 172.21.15.7:0/1631098947 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f50b4098d10 0x7f50b412f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.774+0000 7f50b8e9d700 1 --2- 172.21.15.7:0/1631098947 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f50b4098910 0x7f50b412f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.774+0000 7f50b8e9d700 1 -- 172.21.15.7:0/1631098947 >> 172.21.15.7:0/1631098947 conn(0x7f50b4071830 msgr2=0x7f50b40a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:54.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.775+0000 7f50b8e9d700 1 -- 172.21.15.7:0/1631098947 shutdown_connections 2023-09-30T01:59:54.776 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.775+0000 7f50b8e9d700 1 -- 172.21.15.7:0/1631098947 wait complete. 2023-09-30T01:59:54.776 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-09-30T01:59:54.786 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2023-09-30T01:59:54.328777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":32644,"kb_used_data":1876,"kb_used_omap":0,"kb_used_meta":30720,"kb_avail":566198396,"statfs":{"total":579820584960,"available":579787157504,"internally_reserved":0,"allocated":1921024,"data_stored":1316285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31457280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002591"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.042899+0000","last_change":"2023-09-30T01:54:47.042422+0000","last_active":"2023-09-30T01:54:47.042899+0000","last_peered":"2023-09-30T01:54:47.042899+0000","last_clean":"2023-09-30T01:54:47.042899+0000","last_became_active":"2023-09-30T01:54:47.042183+0000","last_became_peered":"2023-09-30T01:54:47.042183+0000","last_unstale":"2023-09-30T01:54:47.042899+0000","last_undegraded":"2023-09-30T01:54:47.042899+0000","last_fullsized":"2023-09-30T01:54:47.042899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:14:57.725898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.897849+0000","last_change":"2023-09-30T01:54:46.896909+0000","last_active":"2023-09-30T01:54:46.897849+0000","last_peered":"2023-09-30T01:54:46.897849+0000","last_clean":"2023-09-30T01:54:46.897849+0000","last_became_active":"2023-09-30T01:54:46.896620+0000","last_became_peered":"2023-09-30T01:54:46.896620+0000","last_unstale":"2023-09-30T01:54:46.897849+0000","last_undegraded":"2023-09-30T01:54:46.897849+0000","last_fullsized":"2023-09-30T01:54:46.897849+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:28:25.053882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.042370+0000","last_change":"2023-09-30T01:54:47.041973+0000","last_active":"2023-09-30T01:54:47.042370+0000","last_peered":"2023-09-30T01:54:47.042370+0000","last_clean":"2023-09-30T01:54:47.042370+0000","last_became_active":"2023-09-30T01:54:47.041671+0000","last_became_peered":"2023-09-30T01:54:47.041671+0000","last_unstale":"2023-09-30T01:54:47.042370+0000","last_undegraded":"2023-09-30T01:54:47.042370+0000","last_fullsized":"2023-09-30T01:54:47.042370+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:08:29.771169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.896726+0000","last_change":"2023-09-30T01:54:46.896432+0000","last_active":"2023-09-30T01:54:46.896726+0000","last_peered":"2023-09-30T01:54:46.896726+0000","last_clean":"2023-09-30T01:54:46.896726+0000","last_became_active":"2023-09-30T01:54:46.896191+0000","last_became_peered":"2023-09-30T01:54:46.896191+0000","last_unstale":"2023-09-30T01:54:46.896726+0000","last_undegraded":"2023-09-30T01:54:46.896726+0000","last_fullsized":"2023-09-30T01:54:46.896726+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:36:51.492282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":160,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.202781+0000","last_change":"2023-09-30T01:54:45.896629+0000","last_active":"2023-09-30T01:54:47.202781+0000","last_peered":"2023-09-30T01:54:47.202781+0000","last_clean":"2023-09-30T01:54:47.202781+0000","last_became_active":"2023-09-30T01:54:45.896315+0000","last_became_peered":"2023-09-30T01:54:45.896315+0000","last_unstale":"2023-09-30T01:54:47.202781+0000","last_undegraded":"2023-09-30T01:54:47.202781+0000","last_fullsized":"2023-09-30T01:54:47.202781+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:44.377605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:44.377605+0000","last_clean_scrub_stamp":"2023-09-30T01:54:44.377605+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T13:48:02.317525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.899133+0000","last_change":"2023-09-30T01:54:46.898655+0000","last_active":"2023-09-30T01:54:46.899133+0000","last_peered":"2023-09-30T01:54:46.899133+0000","last_clean":"2023-09-30T01:54:46.899133+0000","last_became_active":"2023-09-30T01:54:46.898402+0000","last_became_peered":"2023-09-30T01:54:46.898402+0000","last_unstale":"2023-09-30T01:54:46.899133+0000","last_undegraded":"2023-09-30T01:54:46.899133+0000","last_fullsized":"2023-09-30T01:54:46.899133+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:07:41.886018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.897962+0000","last_change":"2023-09-30T01:54:46.896375+0000","last_active":"2023-09-30T01:54:46.897962+0000","last_peered":"2023-09-30T01:54:46.897962+0000","last_clean":"2023-09-30T01:54:46.897962+0000","last_became_active":"2023-09-30T01:54:46.895973+0000","last_became_peered":"2023-09-30T01:54:46.895973+0000","last_unstale":"2023-09-30T01:54:46.897962+0000","last_undegraded":"2023-09-30T01:54:46.897962+0000","last_fullsized":"2023-09-30T01:54:46.897962+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:52:22.627216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.898698+0000","last_change":"2023-09-30T01:54:46.898322+0000","last_active":"2023-09-30T01:54:46.898698+0000","last_peered":"2023-09-30T01:54:46.898698+0000","last_clean":"2023-09-30T01:54:46.898698+0000","last_became_active":"2023-09-30T01:54:46.898101+0000","last_became_peered":"2023-09-30T01:54:46.898101+0000","last_unstale":"2023-09-30T01:54:46.898698+0000","last_undegraded":"2023-09-30T01:54:46.898698+0000","last_fullsized":"2023-09-30T01:54:46.898698+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:45:31.230458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.042677+0000","last_change":"2023-09-30T01:54:47.042174+0000","last_active":"2023-09-30T01:54:47.042677+0000","last_peered":"2023-09-30T01:54:47.042677+0000","last_clean":"2023-09-30T01:54:47.042677+0000","last_became_active":"2023-09-30T01:54:47.041914+0000","last_became_peered":"2023-09-30T01:54:47.041914+0000","last_unstale":"2023-09-30T01:54:47.042677+0000","last_undegraded":"2023-09-30T01:54:47.042677+0000","last_fullsized":"2023-09-30T01:54:47.042677+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:16:46.356909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5076,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366764,"statfs":{"total":96636764160,"available":96631566336,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52500000000000002,"15min":0.54300000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.69199999999999995,"5min":0.747,"15min":0.747},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53500000000000003,"15min":0.56100000000000005},"min":{"1min":0.40799999999999997,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66400000000000003,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.65100000000000002}]},{"osd":1,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.51500000000000001,"15min":0.53700000000000003},"min":{"1min":0.376,"5min":0.313,"15min":0.313},"max":{"1min":0.66400000000000003,"5min":0.751,"15min":0.751},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.53500000000000003,"15min":0.56100000000000005},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.629,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.55300000000000005}]},{"osd":2,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55100000000000005,"15min":0.57299999999999995},"min":{"1min":0.42499999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.73899999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55300000000000005,"15min":0.56499999999999995},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72599999999999998,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.57899999999999996}]},{"osd":3,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41999999999999998,"15min":0.443},"min":{"1min":0.29899999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.60499999999999998,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45500000000000002,"15min":0.47899999999999998},"min":{"1min":0.312,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.439}]},{"osd":4,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.45500000000000002,"15min":0.48999999999999999},"min":{"1min":0.311,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.50900000000000001,"15min":0.53500000000000003},"min":{"1min":0.36599999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.76700000000000002,"5min":0.874,"15min":0.874},"last":0.46000000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640319,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5076,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366764,"statfs":{"total":96636764160,"available":96631566336,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:58:51 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.30599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.63800000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.50700000000000001,"15min":0.48499999999999999},"min":{"1min":0.37,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66000000000000003,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.47799999999999998}]},{"osd":1,"last update":"Sat Sep 30 01:58:51 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51000000000000001,"15min":0.48799999999999999},"min":{"1min":0.41999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68200000000000005,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.52500000000000002},"min":{"1min":0.47499999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.623,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.52200000000000002}]},{"osd":2,"last update":"Sat Sep 30 01:58:51 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.52300000000000002,"15min":0.504},"min":{"1min":0.38600000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.64600000000000002,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54700000000000004,"15min":0.52500000000000002},"min":{"1min":0.45700000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.69999999999999996,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.53400000000000003}]},{"osd":3,"last update":"Sat Sep 30 01:58:51 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.309,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.54400000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45300000000000001,"15min":0.45700000000000002},"min":{"1min":0.35599999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.374}]},{"osd":5,"last update":"Sat Sep 30 01:58:51 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45000000000000001,"15min":0.45400000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.65300000000000002,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.498,"15min":0.505},"min":{"1min":0.32700000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66200000000000003,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.35799999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640319,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6108,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365732,"statfs":{"total":96636764160,"available":96630509568,"internally_reserved":0,"allocated":675840,"data_stored":572143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.36799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.70199999999999996,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.53800000000000003,"15min":0.54100000000000004},"min":{"1min":0.35899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.71499999999999997,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.746}]},{"osd":1,"last update":"Sat Sep 30 01:58:53 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51400000000000001,"15min":0.51200000000000001},"min":{"1min":0.36599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.622},{"interface":"front","average":{"1min":0.503,"5min":0.50600000000000001,"15min":0.497},"min":{"1min":0.38400000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.63500000000000001,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.70999999999999996}]},{"osd":2,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.51500000000000001,"15min":0.51100000000000001},"min":{"1min":0.434,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.64100000000000001,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54000000000000004,"15min":0.54800000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69299999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.66800000000000004}]},{"osd":4,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.26900000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.59499999999999997,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.48899999999999999,"15min":0.501},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.60599999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.41499999999999998}]},{"osd":5,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.45800000000000002,"15min":0.47399999999999998},"min":{"1min":0.28599999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65400000000000003,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.441,"5min":0.46999999999999997,"15min":0.48199999999999998},"min":{"1min":0.31900000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.59999999999999998,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.54900000000000004}]}]},{"osd":2,"up_from":11,"seq":47244640320,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5140,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366700,"statfs":{"total":96636764160,"available":96631500800,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35299999999999998},"min":{"1min":0.245,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.47999999999999998,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.39300000000000002,"15min":0.38500000000000001},"min":{"1min":0.28100000000000003,"5min":0.254,"15min":0.254},"max":{"1min":0.54300000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.318}]},{"osd":1,"last update":"Sat Sep 30 01:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36799999999999999,"15min":0.37},"min":{"1min":0.23999999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.55800000000000005,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.374,"15min":0.36199999999999999},"min":{"1min":0.251,"5min":0.222,"15min":0.222},"max":{"1min":0.56799999999999995,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.29999999999999999}]},{"osd":3,"last update":"Sat Sep 30 01:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.43099999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.56599999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.54400000000000004}]},{"osd":4,"last update":"Sat Sep 30 01:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.436,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.82699999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.443},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59599999999999997,"15min":0.58399999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.52100000000000002}]},{"osd":5,"last update":"Sat Sep 30 01:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57999999999999996,"15min":0.56999999999999995},"min":{"1min":0.44700000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61499999999999999,"15min":0.61299999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.93799999999999994,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.53400000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640319,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6104,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365736,"statfs":{"total":96636764160,"available":96630513664,"internally_reserved":0,"allocated":671744,"data_stored":571282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34000000000000002,"15min":0.33400000000000002},"min":{"1min":0.219,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.54300000000000004,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.224},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.39500000000000002,"15min":0.39700000000000002},"min":{"1min":0.26700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.59899999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.30299999999999999}]},{"osd":2,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.253,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.51600000000000001,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42199999999999999,"15min":0.42799999999999999},"min":{"1min":0.29899999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.56499999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.34699999999999998}]},{"osd":3,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.56799999999999995,"15min":0.56399999999999995},"min":{"1min":0.35199999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.73999999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.73299999999999998,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.61599999999999999}]},{"osd":4,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.57599999999999996,"15min":0.57899999999999996},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.78900000000000003,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57499999999999996,"15min":0.57799999999999996},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.77900000000000003,"5min":1.026,"15min":1.026},"last":0.51400000000000001}]},{"osd":5,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57199999999999995,"15min":0.57799999999999996},"min":{"1min":0.40999999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76100000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.746,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.66500000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640321,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5140,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366700,"statfs":{"total":96636764160,"available":96631500800,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:58:52 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.32800000000000001,"15min":0.308},"min":{"1min":0.23799999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.49299999999999999,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.33300000000000002,"15min":0.317},"min":{"1min":0.248,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54600000000000004,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.50900000000000001}]},{"osd":2,"last update":"Sat Sep 30 01:58:52 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.35199999999999998,"15min":0.33500000000000002},"min":{"1min":0.28599999999999998,"5min":0.22,"15min":0.22},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.371,"15min":0.36299999999999999},"min":{"1min":0.26800000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.57099999999999995,"5min":0.62,"15min":0.62},"last":0.48699999999999999}]},{"osd":3,"last update":"Sat Sep 30 01:58:52 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.52900000000000003,"15min":0.51500000000000001},"min":{"1min":0.40799999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71099999999999997,"5min":0.748,"15min":0.748},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53700000000000003,"15min":0.53200000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82899999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.60099999999999998}]},{"osd":4,"last update":"Sat Sep 30 01:58:52 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56899999999999995,"15min":0.56200000000000006},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.44700000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.77700000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.63400000000000001}]},{"osd":5,"last update":"Sat Sep 30 01:58:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56100000000000005,"15min":0.55200000000000005},"min":{"1min":0.39600000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.77100000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.58099999999999996,"15min":0.56799999999999995},"min":{"1min":0.42299999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.78400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:59:54.788 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T01:59:54.788 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:59:54.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.905+0000 7f008b9d9700 10 monclient: get_monmap_and_config 2023-09-30T01:59:54.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.906+0000 7f008b9d9700 10 monclient: build_initial_monmap 2023-09-30T01:59:54.906 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.906+0000 7f008b9d9700 10 monclient: monmap: 2023-09-30T01:59:54.906 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:54.906 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:54.906 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:54.907047+0000 2023-09-30T01:59:54.906 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:54.907047+0000 2023-09-30T01:59:54.906 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:54.906 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:54.907 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:54.907 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:54.907 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:54.907 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:54.907 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:54.907 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:54.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.906+0000 7f008b9d9700 1 Processor -- start 2023-09-30T01:59:54.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.906+0000 7f008b9d9700 1 -- start start 2023-09-30T01:59:54.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.906+0000 7f008b9d9700 10 monclient: init 2023-09-30T01:59:54.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f008b9d9700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:54.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f008b9d9700 10 monclient: _add_conns ranks=[0,1,2,4,3] 2023-09-30T01:59:54.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f008b9d9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0084096a50 addr v1:172.21.15.7:6789/0 2023-09-30T01:59:54.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f008b9d9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f00840a2390 addr v1:172.21.15.50:6789/0 2023-09-30T01:59:54.909 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f008b9d9700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0084097530 0x7f008413a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f008b9d9700 10 monclient(hunting): picked mon.noname-b con 0x7f0084097530 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:54.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f008b9d9700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00840980e0 con 0x7f0084096a50 2023-09-30T01:59:54.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f008b9d9700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0084132eb0 con 0x7f00840a2390 2023-09-30T01:59:54.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f008b9d9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:54.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f008b9d9700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00840991a0 con 0x7f0084097530 2023-09-30T01:59:54.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f008b9d9700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:54.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f008b9d9700 10 monclient(hunting): authenticate will time out at 1483.972266s 2023-09-30T01:59:54.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f0089f76700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0084097530 0x7f008413a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f0089775700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0084096a50 0x7f00841327a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48930/0 (socket says 172.21.15.7:48930) 2023-09-30T01:59:54.910 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f0089775700 1 -- 172.21.15.7:0/3362248618 learned_addr learned my addr 172.21.15.7:0/3362248618 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:59:54.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f0089f76700 10 monclient(hunting): get_auth_request con 0x7f0084097530 auth_method 0 2023-09-30T01:59:54.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f0089f76700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:54.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f0089f76700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:54.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f0089f76700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:54.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f0089f76700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:54.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f0089f76700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:54.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f0089f76700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:54.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f0089f76700 10 monclient(hunting): handle_auth_done global_id 4766 payload 274 2023-09-30T01:59:54.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f0089f76700 1 -- 172.21.15.7:0/3362248618 >> v1:172.21.15.50:6789/0 conn(0x7f00840a2390 legacy=0x7f0084096e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:54.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f0089f76700 1 -- 172.21.15.7:0/3362248618 >> v1:172.21.15.7:6789/0 conn(0x7f0084096a50 legacy=0x7f00841327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:54.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f0089f76700 10 monclient: _finish_hunting 0 2023-09-30T01:59:54.911 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f0089f76700 1 monclient: found mon.noname-b 2023-09-30T01:59:54.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f0089f76700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:54.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f0089f76700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:54.912 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.907+0000 7f0089f76700 1 -- 172.21.15.7:0/3362248618 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0084099480 con 0x7f0084097530 2023-09-30T01:59:54.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.909+0000 7f0089f76700 1 -- 172.21.15.7:0/3362248618 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0084097530 msgr2=0x7f008413a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:54.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.909+0000 7f0089f76700 1 --2- 172.21.15.7:0/3362248618 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0084097530 0x7f008413a730 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0074009910 tx=0x7f00740091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6869b02043e932d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:54.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f0083fff700 1 -- 172.21.15.7:0/3362248618 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007400b9f0 con 0x7f0084097530 2023-09-30T01:59:54.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f0083fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:54.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f0083fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:54.918 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f0083fff700 10 monclient: dump: 2023-09-30T01:59:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:54.919 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f0083fff700 10 monclient: _finish_auth 0 2023-09-30T01:59:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f0083fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:54.919 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f008b9d9700 5 monclient: authenticate success, global_id 4766 2023-09-30T01:59:54.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f008b9d9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:54.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f0083fff700 1 -- 172.21.15.7:0/3362248618 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f007400bb50 con 0x7f0084097530 2023-09-30T01:59:54.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f0083fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:54.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f008b9d9700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:54.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f008b9d9700 4 set_mon_vals no callback set 2023-09-30T01:59:54.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f0083fff700 1 -- 172.21.15.7:0/3362248618 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0074011420 con 0x7f0084097530 2023-09-30T01:59:54.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f0083fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:54.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f0083fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:54.920 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f0083fff700 10 monclient: dump: 2023-09-30T01:59:54.920 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:54.920 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:54.921 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.921 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.921 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:54.921 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:54.921 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:54.921 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:54.921 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:54.921 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:54.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f008b9d9700 10 monclient: shutdown 2023-09-30T01:59:54.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f008b9d9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:54.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f008b9d9700 1 -- 172.21.15.7:0/3362248618 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0084097530 msgr2=0x7f008413a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:54.921 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f008b9d9700 1 --2- 172.21.15.7:0/3362248618 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0084097530 0x7f008413a730 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0074009910 tx=0x7f00740091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f008b9d9700 1 -- 172.21.15.7:0/3362248618 shutdown_connections 2023-09-30T01:59:54.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f008b9d9700 1 --2- 172.21.15.7:0/3362248618 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0084097530 0x7f008413a730 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f008b9d9700 1 -- 172.21.15.7:0/3362248618 >> 172.21.15.7:0/3362248618 conn(0x7f0084070cf0 msgr2=0x7f00840710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:54.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f008b9d9700 1 -- 172.21.15.7:0/3362248618 shutdown_connections 2023-09-30T01:59:54.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.910+0000 7f008b9d9700 1 -- 172.21.15.7:0/3362248618 wait complete. 2023-09-30T01:59:54.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.911+0000 7f008b9d9700 10 monclient: build_initial_monmap 2023-09-30T01:59:54.922 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.911+0000 7f008b9d9700 10 monclient: monmap: 2023-09-30T01:59:54.922 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:54.922 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:54.922 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:54.922 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:54.923 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:54.923 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:54.923 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:54.923 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:54.923 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:54.923 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:54.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.911+0000 7f008b9d9700 1 Processor -- start 2023-09-30T01:59:54.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.911+0000 7f008b9d9700 1 -- start start 2023-09-30T01:59:54.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.911+0000 7f008b9d9700 10 monclient: init 2023-09-30T01:59:54.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f008b9d9700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:54.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f008b9d9700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:59:54.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f008b9d9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0084096a50 0x7f00840a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f008b9d9700 10 monclient(hunting): picked mon.noname-b con 0x7f0084096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:54.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f008b9d9700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0084097530 0x7f008409a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f008b9d9700 10 monclient(hunting): picked mon.noname-c con 0x7f0084097530 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:54.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f008b9d9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00840a2390 0x7f008409a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f008b9d9700 10 monclient(hunting): picked mon.noname-a con 0x7f00840a2390 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:54.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f008b9d9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:54.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f008b9d9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008409e130 con 0x7f00840a2390 2023-09-30T01:59:54.924 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f008b9d9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f008b9d9700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00841334e0 con 0x7f0084096a50 2023-09-30T01:59:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f008b9d9700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f008b9d9700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00840a5230 con 0x7f0084097530 2023-09-30T01:59:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f008b9d9700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f008b9d9700 10 monclient(hunting): authenticate will time out at 1483.977179s 2023-09-30T01:59:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0089f76700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00840a2390 0x7f008409a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0088f74700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0084097530 0x7f008409a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0088f74700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0084097530 0x7f008409a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41300/0 (socket says 172.21.15.7:41300) 2023-09-30T01:59:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0089f76700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00840a2390 0x7f008409a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49314/0 (socket says 172.21.15.7:49314) 2023-09-30T01:59:54.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0088f74700 1 -- 172.21.15.7:0/4229716796 learned_addr learned my addr 172.21.15.7:0/4229716796 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0088f74700 10 monclient(hunting): get_auth_request con 0x7f0084097530 auth_method 0 2023-09-30T01:59:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0088f74700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0088f74700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0088f74700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0089f76700 10 monclient(hunting): get_auth_request con 0x7f00840a2390 auth_method 0 2023-09-30T01:59:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0089f76700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0089f76700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0089f76700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0088f74700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0088f74700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:54.926 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0088f74700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0089775700 1 --2- 172.21.15.7:0/4229716796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0084096a50 0x7f00840a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0089f76700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0089f76700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0089f76700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0089775700 10 monclient(hunting): get_auth_request con 0x7f0084096a50 auth_method 0 2023-09-30T01:59:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0089775700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0089775700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0089775700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0089775700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0089775700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:54.927 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0089775700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0088f74700 10 monclient(hunting): handle_auth_done global_id 4769 payload 824 2023-09-30T01:59:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0088f74700 1 -- 172.21.15.7:0/4229716796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0084096a50 msgr2=0x7f00840a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0088f74700 1 --2- 172.21.15.7:0/4229716796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0084096a50 0x7f00840a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0088f74700 1 -- 172.21.15.7:0/4229716796 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00840a2390 msgr2=0x7f008409a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0088f74700 1 --2- 172.21.15.7:0/4229716796 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00840a2390 0x7f008409a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0088f74700 10 monclient: _finish_hunting 0 2023-09-30T01:59:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0088f74700 1 monclient: found mon.noname-c 2023-09-30T01:59:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0088f74700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0088f74700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:59:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0088f74700 1 -- 172.21.15.7:0/4229716796 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0074009420 con 0x7f0084097530 2023-09-30T01:59:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.912+0000 7f0089f76700 1 --2- 172.21.15.7:0/4229716796 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00840a2390 0x7f008409a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:54.928 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.914+0000 7f0088f74700 1 -- 172.21.15.7:0/4229716796 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0084097530 msgr2=0x7f008409a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.914+0000 7f0088f74700 1 --2- 172.21.15.7:0/4229716796 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0084097530 0x7f008409a490 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f007800c600 tx=0x7f0078007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.915+0000 7f00817fa700 1 -- 172.21.15.7:0/4229716796 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00780117a0 con 0x7f0084097530 2023-09-30T01:59:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.915+0000 7f00817fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.915+0000 7f00817fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:59:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.915+0000 7f00817fa700 10 monclient: dump: 2023-09-30T01:59:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:54.929 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:54.930 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:54.930 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:54.930 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:54.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.915+0000 7f00817fa700 10 monclient: _finish_auth 0 2023-09-30T01:59:54.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.915+0000 7f00817fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:54.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.915+0000 7f00817fa700 1 -- 172.21.15.7:0/4229716796 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0078011900 con 0x7f0084097530 2023-09-30T01:59:54.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.915+0000 7f008b9d9700 5 monclient: authenticate success, global_id 4769 2023-09-30T01:59:54.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.915+0000 7f008b9d9700 10 monclient: _renew_subs 2023-09-30T01:59:54.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.915+0000 7f008b9d9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:54.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.915+0000 7f008b9d9700 1 -- 172.21.15.7:0/4229716796 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f008409e7a0 con 0x7f0084097530 2023-09-30T01:59:54.930 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.915+0000 7f00817fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.915+0000 7f00817fa700 1 -- 172.21.15.7:0/4229716796 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0078011cc0 con 0x7f0084097530 2023-09-30T01:59:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.915+0000 7f00817fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.915+0000 7f00817fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.915+0000 7f00817fa700 10 monclient: dump: 2023-09-30T01:59:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:54.931 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:54.932 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:54.932 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:54.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.915+0000 7f00837fe700 4 set_mon_vals no callback set 2023-09-30T01:59:54.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.915+0000 7f008b9d9700 10 monclient: _renew_subs 2023-09-30T01:59:54.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.915+0000 7f008b9d9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:54.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.915+0000 7f008b9d9700 1 -- 172.21.15.7:0/4229716796 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00840a1af0 con 0x7f0084097530 2023-09-30T01:59:54.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.916+0000 7f008b9d9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:54.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.916+0000 7f008b9d9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:54.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.916+0000 7f008b9d9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:54.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.916+0000 7f00817fa700 1 -- 172.21.15.7:0/4229716796 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f0078012630 con 0x7f0084097530 2023-09-30T01:59:54.932 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.916+0000 7f008b9d9700 1 -- 172.21.15.7:0/4229716796 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f008409df90 con 0x7f0084097530 2023-09-30T01:59:54.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.916+0000 7f00817fa700 1 --2- 172.21.15.7:0/4229716796 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f00700357e0 0x7f0070037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:54.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.916+0000 7f00817fa700 1 -- 172.21.15.7:0/4229716796 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f00780479a0 con 0x7f0084097530 2023-09-30T01:59:54.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.916+0000 7f0089775700 1 --2- 172.21.15.7:0/4229716796 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f00700357e0 0x7f0070037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:54.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.916+0000 7f0089775700 10 monclient: get_auth_request con 0x7f00700357e0 auth_method 0 2023-09-30T01:59:54.933 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.919+0000 7f0089775700 1 --2- 172.21.15.7:0/4229716796 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f00700357e0 0x7f0070037c90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f007c006e80 tx=0x7f007c0053c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:54.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.920+0000 7f00817fa700 1 -- 172.21.15.7:0/4229716796 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f008409df90 con 0x7f0084097530 2023-09-30T01:59:54.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.920+0000 7f00817fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:54.934 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:54.920+0000 7f00817fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:55.107 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.107+0000 7f008b9d9700 1 -- 172.21.15.7:0/4229716796 --> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f008409df90 con 0x7f00700357e0 2023-09-30T01:59:55.109 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.110+0000 7f00817fa700 1 -- 172.21.15.7:0/4229716796 <== mgr.4105 v2:172.21.15.50:6824/86200 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53791 (secure 0 0 0) 0x7f008409df90 con 0x7f00700357e0 2023-09-30T01:59:55.110 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:59:55.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.111+0000 7f008b9d9700 1 -- 172.21.15.7:0/4229716796 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f00700357e0 msgr2=0x7f0070037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:55.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.111+0000 7f008b9d9700 1 --2- 172.21.15.7:0/4229716796 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f00700357e0 0x7f0070037c90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f007c006e80 tx=0x7f007c0053c0 comp rx=0 tx=0).stop 2023-09-30T01:59:55.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.111+0000 7f008b9d9700 10 monclient: shutdown 2023-09-30T01:59:55.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.111+0000 7f008b9d9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:55.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.111+0000 7f008b9d9700 1 -- 172.21.15.7:0/4229716796 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0084097530 msgr2=0x7f008409a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:55.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.111+0000 7f008b9d9700 1 --2- 172.21.15.7:0/4229716796 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0084097530 0x7f008409a490 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f007800c600 tx=0x7f0078007f60 comp rx=0 tx=0).stop 2023-09-30T01:59:55.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.111+0000 7f008b9d9700 1 -- 172.21.15.7:0/4229716796 shutdown_connections 2023-09-30T01:59:55.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.111+0000 7f008b9d9700 1 --2- 172.21.15.7:0/4229716796 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f00700357e0 0x7f0070037c90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:55.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.111+0000 7f008b9d9700 1 --2- 172.21.15.7:0/4229716796 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00840a2390 0x7f008409a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:55.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.111+0000 7f008b9d9700 1 --2- 172.21.15.7:0/4229716796 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0084096a50 0x7f00840a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:55.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.111+0000 7f008b9d9700 1 --2- 172.21.15.7:0/4229716796 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f0084097530 0x7f008409a490 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:55.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.111+0000 7f008b9d9700 1 -- 172.21.15.7:0/4229716796 >> 172.21.15.7:0/4229716796 conn(0x7f0084070cf0 msgr2=0x7f008413a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:55.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.111+0000 7f008b9d9700 1 -- 172.21.15.7:0/4229716796 shutdown_connections 2023-09-30T01:59:55.113 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.111+0000 7f008b9d9700 1 -- 172.21.15.7:0/4229716796 wait complete. 2023-09-30T01:59:55.113 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-09-30T01:59:55.123 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2023-09-30T01:59:54.328777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":32644,"kb_used_data":1876,"kb_used_omap":0,"kb_used_meta":30720,"kb_avail":566198396,"statfs":{"total":579820584960,"available":579787157504,"internally_reserved":0,"allocated":1921024,"data_stored":1316285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31457280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002591"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.042899+0000","last_change":"2023-09-30T01:54:47.042422+0000","last_active":"2023-09-30T01:54:47.042899+0000","last_peered":"2023-09-30T01:54:47.042899+0000","last_clean":"2023-09-30T01:54:47.042899+0000","last_became_active":"2023-09-30T01:54:47.042183+0000","last_became_peered":"2023-09-30T01:54:47.042183+0000","last_unstale":"2023-09-30T01:54:47.042899+0000","last_undegraded":"2023-09-30T01:54:47.042899+0000","last_fullsized":"2023-09-30T01:54:47.042899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:14:57.725898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.897849+0000","last_change":"2023-09-30T01:54:46.896909+0000","last_active":"2023-09-30T01:54:46.897849+0000","last_peered":"2023-09-30T01:54:46.897849+0000","last_clean":"2023-09-30T01:54:46.897849+0000","last_became_active":"2023-09-30T01:54:46.896620+0000","last_became_peered":"2023-09-30T01:54:46.896620+0000","last_unstale":"2023-09-30T01:54:46.897849+0000","last_undegraded":"2023-09-30T01:54:46.897849+0000","last_fullsized":"2023-09-30T01:54:46.897849+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:28:25.053882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.042370+0000","last_change":"2023-09-30T01:54:47.041973+0000","last_active":"2023-09-30T01:54:47.042370+0000","last_peered":"2023-09-30T01:54:47.042370+0000","last_clean":"2023-09-30T01:54:47.042370+0000","last_became_active":"2023-09-30T01:54:47.041671+0000","last_became_peered":"2023-09-30T01:54:47.041671+0000","last_unstale":"2023-09-30T01:54:47.042370+0000","last_undegraded":"2023-09-30T01:54:47.042370+0000","last_fullsized":"2023-09-30T01:54:47.042370+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:08:29.771169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.896726+0000","last_change":"2023-09-30T01:54:46.896432+0000","last_active":"2023-09-30T01:54:46.896726+0000","last_peered":"2023-09-30T01:54:46.896726+0000","last_clean":"2023-09-30T01:54:46.896726+0000","last_became_active":"2023-09-30T01:54:46.896191+0000","last_became_peered":"2023-09-30T01:54:46.896191+0000","last_unstale":"2023-09-30T01:54:46.896726+0000","last_undegraded":"2023-09-30T01:54:46.896726+0000","last_fullsized":"2023-09-30T01:54:46.896726+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:36:51.492282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":160,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.202781+0000","last_change":"2023-09-30T01:54:45.896629+0000","last_active":"2023-09-30T01:54:47.202781+0000","last_peered":"2023-09-30T01:54:47.202781+0000","last_clean":"2023-09-30T01:54:47.202781+0000","last_became_active":"2023-09-30T01:54:45.896315+0000","last_became_peered":"2023-09-30T01:54:45.896315+0000","last_unstale":"2023-09-30T01:54:47.202781+0000","last_undegraded":"2023-09-30T01:54:47.202781+0000","last_fullsized":"2023-09-30T01:54:47.202781+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:44.377605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:44.377605+0000","last_clean_scrub_stamp":"2023-09-30T01:54:44.377605+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T13:48:02.317525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.899133+0000","last_change":"2023-09-30T01:54:46.898655+0000","last_active":"2023-09-30T01:54:46.899133+0000","last_peered":"2023-09-30T01:54:46.899133+0000","last_clean":"2023-09-30T01:54:46.899133+0000","last_became_active":"2023-09-30T01:54:46.898402+0000","last_became_peered":"2023-09-30T01:54:46.898402+0000","last_unstale":"2023-09-30T01:54:46.899133+0000","last_undegraded":"2023-09-30T01:54:46.899133+0000","last_fullsized":"2023-09-30T01:54:46.899133+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:07:41.886018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.897962+0000","last_change":"2023-09-30T01:54:46.896375+0000","last_active":"2023-09-30T01:54:46.897962+0000","last_peered":"2023-09-30T01:54:46.897962+0000","last_clean":"2023-09-30T01:54:46.897962+0000","last_became_active":"2023-09-30T01:54:46.895973+0000","last_became_peered":"2023-09-30T01:54:46.895973+0000","last_unstale":"2023-09-30T01:54:46.897962+0000","last_undegraded":"2023-09-30T01:54:46.897962+0000","last_fullsized":"2023-09-30T01:54:46.897962+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:52:22.627216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.898698+0000","last_change":"2023-09-30T01:54:46.898322+0000","last_active":"2023-09-30T01:54:46.898698+0000","last_peered":"2023-09-30T01:54:46.898698+0000","last_clean":"2023-09-30T01:54:46.898698+0000","last_became_active":"2023-09-30T01:54:46.898101+0000","last_became_peered":"2023-09-30T01:54:46.898101+0000","last_unstale":"2023-09-30T01:54:46.898698+0000","last_undegraded":"2023-09-30T01:54:46.898698+0000","last_fullsized":"2023-09-30T01:54:46.898698+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:45:31.230458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.042677+0000","last_change":"2023-09-30T01:54:47.042174+0000","last_active":"2023-09-30T01:54:47.042677+0000","last_peered":"2023-09-30T01:54:47.042677+0000","last_clean":"2023-09-30T01:54:47.042677+0000","last_became_active":"2023-09-30T01:54:47.041914+0000","last_became_peered":"2023-09-30T01:54:47.041914+0000","last_unstale":"2023-09-30T01:54:47.042677+0000","last_undegraded":"2023-09-30T01:54:47.042677+0000","last_fullsized":"2023-09-30T01:54:47.042677+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:16:46.356909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5076,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366764,"statfs":{"total":96636764160,"available":96631566336,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52500000000000002,"15min":0.54300000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.69199999999999995,"5min":0.747,"15min":0.747},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53500000000000003,"15min":0.56100000000000005},"min":{"1min":0.40799999999999997,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66400000000000003,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.65100000000000002}]},{"osd":1,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.51500000000000001,"15min":0.53700000000000003},"min":{"1min":0.376,"5min":0.313,"15min":0.313},"max":{"1min":0.66400000000000003,"5min":0.751,"15min":0.751},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.53500000000000003,"15min":0.56100000000000005},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.629,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.55300000000000005}]},{"osd":2,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55100000000000005,"15min":0.57299999999999995},"min":{"1min":0.42499999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.73899999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55300000000000005,"15min":0.56499999999999995},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72599999999999998,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.57899999999999996}]},{"osd":3,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41999999999999998,"15min":0.443},"min":{"1min":0.29899999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.60499999999999998,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45500000000000002,"15min":0.47899999999999998},"min":{"1min":0.312,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.439}]},{"osd":4,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.45500000000000002,"15min":0.48999999999999999},"min":{"1min":0.311,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.50900000000000001,"15min":0.53500000000000003},"min":{"1min":0.36599999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.76700000000000002,"5min":0.874,"15min":0.874},"last":0.46000000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640319,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5076,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366764,"statfs":{"total":96636764160,"available":96631566336,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:58:51 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.30599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.63800000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.50700000000000001,"15min":0.48499999999999999},"min":{"1min":0.37,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.66000000000000003,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.47799999999999998}]},{"osd":1,"last update":"Sat Sep 30 01:58:51 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51000000000000001,"15min":0.48799999999999999},"min":{"1min":0.41999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68200000000000005,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.53500000000000003,"15min":0.52500000000000002},"min":{"1min":0.47499999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.623,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.52200000000000002}]},{"osd":2,"last update":"Sat Sep 30 01:58:51 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.52300000000000002,"15min":0.504},"min":{"1min":0.38600000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.64600000000000002,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54700000000000004,"15min":0.52500000000000002},"min":{"1min":0.45700000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.69999999999999996,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.53400000000000003}]},{"osd":3,"last update":"Sat Sep 30 01:58:51 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.309,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.54400000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45300000000000001,"15min":0.45700000000000002},"min":{"1min":0.35599999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.374}]},{"osd":5,"last update":"Sat Sep 30 01:58:51 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45000000000000001,"15min":0.45400000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.65300000000000002,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.498,"15min":0.505},"min":{"1min":0.32700000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66200000000000003,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.35799999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640319,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6108,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365732,"statfs":{"total":96636764160,"available":96630509568,"internally_reserved":0,"allocated":675840,"data_stored":572143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.36799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.70199999999999996,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.53800000000000003,"15min":0.54100000000000004},"min":{"1min":0.35899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.71499999999999997,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.746}]},{"osd":1,"last update":"Sat Sep 30 01:58:53 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51400000000000001,"15min":0.51200000000000001},"min":{"1min":0.36599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.622},{"interface":"front","average":{"1min":0.503,"5min":0.50600000000000001,"15min":0.497},"min":{"1min":0.38400000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.63500000000000001,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.70999999999999996}]},{"osd":2,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.51500000000000001,"15min":0.51100000000000001},"min":{"1min":0.434,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.64100000000000001,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54000000000000004,"15min":0.54800000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69299999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.66800000000000004}]},{"osd":4,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.26900000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.59499999999999997,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.48899999999999999,"15min":0.501},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.60599999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.41499999999999998}]},{"osd":5,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.45800000000000002,"15min":0.47399999999999998},"min":{"1min":0.28599999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65400000000000003,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.441,"5min":0.46999999999999997,"15min":0.48199999999999998},"min":{"1min":0.31900000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.59999999999999998,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.54900000000000004}]}]},{"osd":2,"up_from":11,"seq":47244640320,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5140,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366700,"statfs":{"total":96636764160,"available":96631500800,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35299999999999998},"min":{"1min":0.245,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.47999999999999998,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.39300000000000002,"15min":0.38500000000000001},"min":{"1min":0.28100000000000003,"5min":0.254,"15min":0.254},"max":{"1min":0.54300000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.318}]},{"osd":1,"last update":"Sat Sep 30 01:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36799999999999999,"15min":0.37},"min":{"1min":0.23999999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.55800000000000005,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.374,"15min":0.36199999999999999},"min":{"1min":0.251,"5min":0.222,"15min":0.222},"max":{"1min":0.56799999999999995,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.29999999999999999}]},{"osd":3,"last update":"Sat Sep 30 01:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.43099999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.56599999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.54400000000000004}]},{"osd":4,"last update":"Sat Sep 30 01:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.436,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.82699999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.443},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59599999999999997,"15min":0.58399999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.52100000000000002}]},{"osd":5,"last update":"Sat Sep 30 01:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57999999999999996,"15min":0.56999999999999995},"min":{"1min":0.44700000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61499999999999999,"15min":0.61299999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.93799999999999994,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.53400000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640319,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6104,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365736,"statfs":{"total":96636764160,"available":96630513664,"internally_reserved":0,"allocated":671744,"data_stored":571282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34000000000000002,"15min":0.33400000000000002},"min":{"1min":0.219,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.54300000000000004,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.224},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.39500000000000002,"15min":0.39700000000000002},"min":{"1min":0.26700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.59899999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.30299999999999999}]},{"osd":2,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.253,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.51600000000000001,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42199999999999999,"15min":0.42799999999999999},"min":{"1min":0.29899999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.56499999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.34699999999999998}]},{"osd":3,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.56799999999999995,"15min":0.56399999999999995},"min":{"1min":0.35199999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.73999999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.55400000000000005,"15min":0.55800000000000005},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.73299999999999998,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.61599999999999999}]},{"osd":4,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.57599999999999996,"15min":0.57899999999999996},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.78900000000000003,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57499999999999996,"15min":0.57799999999999996},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.77900000000000003,"5min":1.026,"15min":1.026},"last":0.51400000000000001}]},{"osd":5,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57199999999999995,"15min":0.57799999999999996},"min":{"1min":0.40999999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76100000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.746,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.66500000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640321,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5140,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366700,"statfs":{"total":96636764160,"available":96631500800,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:58:52 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.32800000000000001,"15min":0.308},"min":{"1min":0.23799999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.49299999999999999,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.33300000000000002,"15min":0.317},"min":{"1min":0.248,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54600000000000004,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.50900000000000001}]},{"osd":2,"last update":"Sat Sep 30 01:58:52 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.35199999999999998,"15min":0.33500000000000002},"min":{"1min":0.28599999999999998,"5min":0.22,"15min":0.22},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.371,"15min":0.36299999999999999},"min":{"1min":0.26800000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.57099999999999995,"5min":0.62,"15min":0.62},"last":0.48699999999999999}]},{"osd":3,"last update":"Sat Sep 30 01:58:52 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.52900000000000003,"15min":0.51500000000000001},"min":{"1min":0.40799999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71099999999999997,"5min":0.748,"15min":0.748},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53700000000000003,"15min":0.53200000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82899999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.60099999999999998}]},{"osd":4,"last update":"Sat Sep 30 01:58:52 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56899999999999995,"15min":0.56200000000000006},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.44700000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.77700000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.63400000000000001}]},{"osd":5,"last update":"Sat Sep 30 01:58:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56100000000000005,"15min":0.55200000000000005},"min":{"1min":0.39600000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.77100000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.58099999999999996,"15min":0.56799999999999995},"min":{"1min":0.42299999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.78400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T01:59:55.125 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T01:59:55.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.243+0000 7f5396ada700 10 monclient: get_monmap_and_config 2023-09-30T01:59:55.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.243+0000 7f5396ada700 10 monclient: build_initial_monmap 2023-09-30T01:59:55.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.243+0000 7f5396ada700 10 monclient: monmap: 2023-09-30T01:59:55.244 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:55.244 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:55.244 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:55.244954+0000 2023-09-30T01:59:55.244 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:55.244954+0000 2023-09-30T01:59:55.244 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:55.244 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:55.244 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:55.244 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:55.245 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:55.245 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:55.245 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:55.245 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:55.251 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.244+0000 7f5396ada700 1 Processor -- start 2023-09-30T01:59:55.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.244+0000 7f5396ada700 1 -- start start 2023-09-30T01:59:55.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.244+0000 7f5396ada700 10 monclient: init 2023-09-30T01:59:55.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5396ada700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:55.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5396ada700 10 monclient: _add_conns ranks=[3,2,4,1,0] 2023-09-30T01:59:55.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5396ada700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5390096a50 0x7f53901327a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:55.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5396ada700 10 monclient(hunting): picked mon.noname-a con 0x7f5390096a50 addr v2:172.21.15.7:3300/0 2023-09-30T01:59:55.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5396ada700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f53900a2390 0x7f5390096e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:55.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5396ada700 10 monclient(hunting): picked mon.noname-b con 0x7f53900a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:55.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5396ada700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f5390097360 0x7f539012ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:55.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5396ada700 10 monclient(hunting): picked mon.noname-c con 0x7f5390097360 addr v2:172.21.15.50:3300/0 2023-09-30T01:59:55.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5396ada700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:55.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5396ada700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53900980e0 con 0x7f5390096a50 2023-09-30T01:59:55.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5396ada700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:55.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5396ada700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53900991a0 con 0x7f5390097360 2023-09-30T01:59:55.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5396ada700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:55.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5396ada700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5390099480 con 0x7f53900a2390 2023-09-30T01:59:55.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5396ada700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:55.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5396ada700 10 monclient(hunting): authenticate will time out at 1484.310119s 2023-09-30T01:59:55.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5394876700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5390096a50 0x7f53901327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:55.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f538ffff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f53900a2390 0x7f5390096e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:55.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f538ffff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f53900a2390 0x7f5390096e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41314/0 (socket says 172.21.15.7:41314) 2023-09-30T01:59:55.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5394876700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f5390096a50 0x7f53901327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49320/0 (socket says 172.21.15.7:49320) 2023-09-30T01:59:55.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f538ffff700 1 -- 172.21.15.7:0/3295123925 learned_addr learned my addr 172.21.15.7:0/3295123925 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:55.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f538ffff700 10 monclient(hunting): get_auth_request con 0x7f53900a2390 auth_method 0 2023-09-30T01:59:55.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f538ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:55.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f538ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:55.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f538ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:55.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5394876700 10 monclient(hunting): get_auth_request con 0x7f5390096a50 auth_method 0 2023-09-30T01:59:55.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5394876700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:55.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5394876700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:55.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5394876700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:55.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f538ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:55.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f538ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:55.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f538ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:55.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5394876700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:55.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5394876700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:55.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5394876700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:55.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5395077700 1 --2- 172.21.15.7:0/3295123925 >> v2:172.21.15.50:3300/0 conn(0x7f5390097360 0x7f539012ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:55.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f538ffff700 10 monclient(hunting): handle_auth_done global_id 4772 payload 274 2023-09-30T01:59:55.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f538ffff700 1 -- 172.21.15.7:0/3295123925 >> v2:172.21.15.50:3300/0 conn(0x7f5390097360 msgr2=0x7f539012ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:55.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f538ffff700 1 --2- 172.21.15.7:0/3295123925 >> v2:172.21.15.50:3300/0 conn(0x7f5390097360 0x7f539012ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:55.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f538ffff700 1 -- 172.21.15.7:0/3295123925 >> v2:172.21.15.7:3300/0 conn(0x7f5390096a50 msgr2=0x7f53901327a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:55.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f538ffff700 1 --2- 172.21.15.7:0/3295123925 >> v2:172.21.15.7:3300/0 conn(0x7f5390096a50 0x7f53901327a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:55.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f538ffff700 10 monclient: _finish_hunting 0 2023-09-30T01:59:55.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f538ffff700 1 monclient: found mon.noname-b 2023-09-30T01:59:55.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f538ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:55.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f538ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:55.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f538ffff700 1 -- 172.21.15.7:0/3295123925 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5390097bf0 con 0x7f53900a2390 2023-09-30T01:59:55.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5395077700 10 monclient: get_auth_request con 0x7f5390097360 auth_method 0 2023-09-30T01:59:55.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5395077700 1 --2- 172.21.15.7:0/3295123925 >> v2:172.21.15.50:3300/0 conn(0x7f5390097360 0x7f539012ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:59:55.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.245+0000 7f5394876700 1 --2- 172.21.15.7:0/3295123925 >> v2:172.21.15.7:3300/0 conn(0x7f5390096a50 0x7f53901327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:55.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.247+0000 7f538ffff700 1 -- 172.21.15.7:0/3295123925 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f53900a2390 msgr2=0x7f5390096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:55.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.247+0000 7f538ffff700 1 --2- 172.21.15.7:0/3295123925 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f53900a2390 0x7f5390096e20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f53900a2010 tx=0x7f537c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80fab7b2147b7f4c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:55.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.247+0000 7f538f7fe700 1 -- 172.21.15.7:0/3295123925 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537c009e70 con 0x7f53900a2390 2023-09-30T01:59:55.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.247+0000 7f538f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:55.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.247+0000 7f538f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:55.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.247+0000 7f538f7fe700 10 monclient: dump: 2023-09-30T01:59:55.257 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:55.257 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:55.257 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:55.257 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:55.257 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:55.257 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:55.257 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:55.257 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:55.257 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:55.258 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:55.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f538f7fe700 10 monclient: _finish_auth 0 2023-09-30T01:59:55.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f538f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:55.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f5396ada700 5 monclient: authenticate success, global_id 4772 2023-09-30T01:59:55.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f5396ada700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:55.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f538f7fe700 1 -- 172.21.15.7:0/3295123925 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f537c00ba70 con 0x7f53900a2390 2023-09-30T01:59:55.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f538f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:55.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f5396ada700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:55.258 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f5396ada700 4 set_mon_vals no callback set 2023-09-30T01:59:55.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f538f7fe700 1 -- 172.21.15.7:0/3295123925 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537c00be30 con 0x7f53900a2390 2023-09-30T01:59:55.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f538f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:55.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f538f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:55.259 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f538f7fe700 10 monclient: dump: 2023-09-30T01:59:55.259 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:55.259 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:55.259 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:55.259 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:55.259 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:55.259 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:55.260 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:55.260 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:55.260 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:55.260 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:55.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f5396ada700 10 monclient: shutdown 2023-09-30T01:59:55.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f5396ada700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:55.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f5396ada700 1 -- 172.21.15.7:0/3295123925 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f53900a2390 msgr2=0x7f5390096e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:55.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f5396ada700 1 --2- 172.21.15.7:0/3295123925 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f53900a2390 0x7f5390096e20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f53900a2010 tx=0x7f537c009250 comp rx=0 tx=0).stop 2023-09-30T01:59:55.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f5396ada700 1 -- 172.21.15.7:0/3295123925 shutdown_connections 2023-09-30T01:59:55.260 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f5396ada700 1 --2- 172.21.15.7:0/3295123925 >> v2:172.21.15.50:3300/0 conn(0x7f5390097360 0x7f539012ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:55.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f5396ada700 1 --2- 172.21.15.7:0/3295123925 >> v2:172.21.15.7:3300/0 conn(0x7f5390096a50 0x7f53901327a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:55.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f5396ada700 1 --2- 172.21.15.7:0/3295123925 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f53900a2390 0x7f5390096e20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:55.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f5396ada700 1 -- 172.21.15.7:0/3295123925 >> 172.21.15.7:0/3295123925 conn(0x7f5390070cf0 msgr2=0x7f53900710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:55.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f5396ada700 1 -- 172.21.15.7:0/3295123925 shutdown_connections 2023-09-30T01:59:55.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.248+0000 7f5396ada700 1 -- 172.21.15.7:0/3295123925 wait complete. 2023-09-30T01:59:55.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.249+0000 7f5396ada700 10 monclient: build_initial_monmap 2023-09-30T01:59:55.261 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.249+0000 7f5396ada700 10 monclient: monmap: 2023-09-30T01:59:55.261 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:55.261 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:55.261 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:55.262 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:55.262 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:55.262 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:55.262 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:55.262 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:55.262 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:55.262 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:55.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.249+0000 7f5396ada700 1 Processor -- start 2023-09-30T01:59:55.262 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.249+0000 7f5396ada700 1 -- start start 2023-09-30T01:59:55.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.249+0000 7f5396ada700 10 monclient: init 2023-09-30T01:59:55.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.249+0000 7f5396ada700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:55.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5396ada700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T01:59:55.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5396ada700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5390096a50 0x7f53900a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:55.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5396ada700 10 monclient(hunting): picked mon.noname-a con 0x7f5390096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:55.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5396ada700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5390097360 0x7f539011c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:55.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5396ada700 10 monclient(hunting): picked mon.noname-b con 0x7f5390097360 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:55.263 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5396ada700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f53900a2390 0x7f539012e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:55.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5396ada700 10 monclient(hunting): picked mon.noname-c con 0x7f53900a2390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:55.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5396ada700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:55.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5396ada700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539011cfb0 con 0x7f5390096a50 2023-09-30T01:59:55.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5396ada700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:55.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5396ada700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53900a17e0 con 0x7f5390097360 2023-09-30T01:59:55.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5396ada700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:55.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5396ada700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539011e530 con 0x7f53900a2390 2023-09-30T01:59:55.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5396ada700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:55.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5396ada700 10 monclient(hunting): authenticate will time out at 1484.315070s 2023-09-30T01:59:55.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5394876700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5390096a50 0x7f53900a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:55.264 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5395077700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f53900a2390 0x7f539012e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:55.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5394876700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5390096a50 0x7f53900a0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49322/0 (socket says 172.21.15.7:49322) 2023-09-30T01:59:55.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5394876700 1 -- 172.21.15.7:0/1573998677 learned_addr learned my addr 172.21.15.7:0/1573998677 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:55.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5394876700 10 monclient(hunting): get_auth_request con 0x7f5390096a50 auth_method 0 2023-09-30T01:59:55.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5394876700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:55.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5394876700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:55.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5394876700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:55.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5395077700 10 monclient(hunting): get_auth_request con 0x7f53900a2390 auth_method 0 2023-09-30T01:59:55.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5395077700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:55.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5395077700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:55.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5395077700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:55.265 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5395077700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:55.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5395077700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:55.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5395077700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:55.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5394876700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:55.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5394876700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:55.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5394876700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:55.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f538ffff700 1 --2- 172.21.15.7:0/1573998677 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5390097360 0x7f539011c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:55.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f538ffff700 10 monclient(hunting): get_auth_request con 0x7f5390097360 auth_method 0 2023-09-30T01:59:55.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f538ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:55.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f538ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:55.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f538ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:55.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5395077700 10 monclient(hunting): handle_auth_done global_id 4775 payload 824 2023-09-30T01:59:55.266 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5395077700 1 -- 172.21.15.7:0/1573998677 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5390097360 msgr2=0x7f539011c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:55.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5395077700 1 --2- 172.21.15.7:0/1573998677 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5390097360 0x7f539011c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:55.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5395077700 1 -- 172.21.15.7:0/1573998677 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5390096a50 msgr2=0x7f53900a0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:55.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5395077700 1 --2- 172.21.15.7:0/1573998677 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5390096a50 0x7f53900a0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:55.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5395077700 10 monclient: _finish_hunting 0 2023-09-30T01:59:55.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5395077700 1 monclient: found mon.noname-c 2023-09-30T01:59:55.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5395077700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:55.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5395077700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:59:55.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f5395077700 1 -- 172.21.15.7:0/1573998677 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5384009420 con 0x7f53900a2390 2023-09-30T01:59:55.267 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.250+0000 7f538ffff700 1 --2- 172.21.15.7:0/1573998677 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5390097360 0x7f539011c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:59:55.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f5395077700 1 -- 172.21.15.7:0/1573998677 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f53900a2390 msgr2=0x7f539012e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:55.269 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f5395077700 1 --2- 172.21.15.7:0/1573998677 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f53900a2390 0x7f539012e100 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f538800cf60 tx=0x7f5388007840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:55.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f538cff9700 1 -- 172.21.15.7:0/1573998677 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538800ae50 con 0x7f53900a2390 2023-09-30T01:59:55.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f538cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:55.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f538cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:59:55.270 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f538cff9700 10 monclient: dump: 2023-09-30T01:59:55.270 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:55.270 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:55.270 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:55.270 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:55.270 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:55.270 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:55.270 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:55.270 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:55.271 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:55.271 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:55.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f538cff9700 10 monclient: _finish_auth 0 2023-09-30T01:59:55.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f538cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:55.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f538cff9700 1 -- 172.21.15.7:0/1573998677 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538800d650 con 0x7f53900a2390 2023-09-30T01:59:55.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f5396ada700 5 monclient: authenticate success, global_id 4775 2023-09-30T01:59:55.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f5396ada700 10 monclient: _renew_subs 2023-09-30T01:59:55.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f5396ada700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:55.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f5396ada700 1 -- 172.21.15.7:0/1573998677 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f537c0094f0 con 0x7f53900a2390 2023-09-30T01:59:55.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f538cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:55.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f538cff9700 1 -- 172.21.15.7:0/1573998677 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538800da10 con 0x7f53900a2390 2023-09-30T01:59:55.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f538cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:55.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f538cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:55.272 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f538cff9700 10 monclient: dump: 2023-09-30T01:59:55.272 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:55.272 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:55.272 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:55.272 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:55.272 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:55.272 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:55.272 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:55.272 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:55.272 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:55.272 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:55.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f538effd700 4 set_mon_vals no callback set 2023-09-30T01:59:55.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f5396ada700 10 monclient: _renew_subs 2023-09-30T01:59:55.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f5396ada700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:55.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.252+0000 7f5396ada700 1 -- 172.21.15.7:0/1573998677 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f539011e810 con 0x7f53900a2390 2023-09-30T01:59:55.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.253+0000 7f538cff9700 1 -- 172.21.15.7:0/1573998677 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f538801e3b0 con 0x7f53900a2390 2023-09-30T01:59:55.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.253+0000 7f5396ada700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:55.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.253+0000 7f5396ada700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:55.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.253+0000 7f5396ada700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:55.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.253+0000 7f5396ada700 1 -- 172.21.15.7:0/1573998677 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5390064c10 con 0x7f53900a2390 2023-09-30T01:59:55.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.253+0000 7f538cff9700 1 --2- 172.21.15.7:0/1573998677 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5374035830 0x7f5374037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:55.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.254+0000 7f538cff9700 1 -- 172.21.15.7:0/1573998677 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f5388048540 con 0x7f53900a2390 2023-09-30T01:59:55.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.254+0000 7f5394876700 1 --2- 172.21.15.7:0/1573998677 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5374035830 0x7f5374037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:55.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.254+0000 7f5394876700 10 monclient: get_auth_request con 0x7f5374035830 auth_method 0 2023-09-30T01:59:55.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.256+0000 7f5394876700 1 --2- 172.21.15.7:0/1573998677 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5374035830 0x7f5374037ce0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5384011310 tx=0x7f53840091a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:55.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.258+0000 7f538cff9700 1 -- 172.21.15.7:0/1573998677 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f53880122a0 con 0x7f53900a2390 2023-09-30T01:59:55.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.258+0000 7f538cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:55.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.258+0000 7f538cff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:55.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.453+0000 7f5396ada700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:59:55.453 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.453+0000 7f5396ada700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:59:55.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.453+0000 7f5396ada700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:55.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.453+0000 7f5396ada700 1 -- 172.21.15.7:0/1573998677 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5390132ce0 con 0x7f53900a2390 2023-09-30T01:59:55.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.454+0000 7f538cff9700 1 -- 172.21.15.7:0/1573998677 <== mon.2 v2:172.21.15.7:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+11236 (secure 0 0 0) 0x7f538801a090 con 0x7f53900a2390 2023-09-30T01:59:55.454 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.454+0000 7f538cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T01:59:55.455 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.454+0000 7f538cff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T01:59:55.455 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T01:59:55.455 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":15,"fsid":"2ea1044d-d984-49b9-baad-7f36bd454a19","created":"2023-09-30T01:54:34.356209+0000","modified":"2023-09-30T01:54:47.895378+0000","last_up_change":"2023-09-30T01:54:43.359934+0000","last_in_change":"2023-09-30T01:54:36.628886+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T01:54:44.376777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T01:54:45.091042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"e44e9247-cb86-4a16-ac1f-f021cdf5108f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":86601},{"type":"v1","addr":"172.21.15.7:6801","nonce":86601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":86601},{"type":"v1","addr":"172.21.15.7:6803","nonce":86601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":86601},{"type":"v1","addr":"172.21.15.7:6807","nonce":86601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":86601},{"type":"v1","addr":"172.21.15.7:6805","nonce":86601}]},"public_addr":"172.21.15.7:6801/86601","cluster_addr":"172.21.15.7:6803/86601","heartbeat_back_addr":"172.21.15.7:6807/86601","heartbeat_front_addr":"172.21.15.7:6805/86601","state":["exists","up"]},{"osd":1,"uuid":"e30730fd-1e49-44bf-bdc9-087e23727df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":86600},{"type":"v1","addr":"172.21.15.7:6817","nonce":86600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":86600},{"type":"v1","addr":"172.21.15.7:6819","nonce":86600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":86600},{"type":"v1","addr":"172.21.15.7:6823","nonce":86600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":86600},{"type":"v1","addr":"172.21.15.7:6821","nonce":86600}]},"public_addr":"172.21.15.7:6817/86600","cluster_addr":"172.21.15.7:6819/86600","heartbeat_back_addr":"172.21.15.7:6823/86600","heartbeat_front_addr":"172.21.15.7:6821/86600","state":["exists","up"]},{"osd":2,"uuid":"b39c552e-f5cc-4e4f-90a3-5aa5f5127a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":86602},{"type":"v1","addr":"172.21.15.7:6809","nonce":86602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":86602},{"type":"v1","addr":"172.21.15.7:6811","nonce":86602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":86602},{"type":"v1","addr":"172.21.15.7:6815","nonce":86602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":86602},{"type":"v1","addr":"172.21.15.7:6813","nonce":86602}]},"public_addr":"172.21.15.7:6809/86602","cluster_addr":"172.21.15.7:6811/86602","heartbeat_back_addr":"172.21.15.7:6815/86602","heartbeat_front_addr":"172.21.15.7:6813/86602","state":["exists","up"]},{"osd":3,"uuid":"15cd89ae-049a-4e1a-8c6b-7d086ca6e1df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":86633},{"type":"v1","addr":"172.21.15.50:6807","nonce":86633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":86633},{"type":"v1","addr":"172.21.15.50:6812","nonce":86633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6819","nonce":86633},{"type":"v1","addr":"172.21.15.50:6821","nonce":86633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6815","nonce":86633},{"type":"v1","addr":"172.21.15.50:6818","nonce":86633}]},"public_addr":"172.21.15.50:6807/86633","cluster_addr":"172.21.15.50:6812/86633","heartbeat_back_addr":"172.21.15.50:6821/86633","heartbeat_front_addr":"172.21.15.50:6818/86633","state":["exists","up"]},{"osd":4,"uuid":"de6773bd-d286-4ab9-b91f-8a90ee3073d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":86632},{"type":"v1","addr":"172.21.15.50:6806","nonce":86632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":86632},{"type":"v1","addr":"172.21.15.50:6814","nonce":86632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":86632},{"type":"v1","addr":"172.21.15.50:6823","nonce":86632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6817","nonce":86632},{"type":"v1","addr":"172.21.15.50:6820","nonce":86632}]},"public_addr":"172.21.15.50:6806/86632","cluster_addr":"172.21.15.50:6814/86632","heartbeat_back_addr":"172.21.15.50:6823/86632","heartbeat_front_addr":"172.21.15.50:6820/86632","state":["exists","up"]},{"osd":5,"uuid":"e6a847ab-276e-4d54-9a8f-6df5b8ab54cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":86634},{"type":"v1","addr":"172.21.15.50:6801","nonce":86634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":86634},{"type":"v1","addr":"172.21.15.50:6805","nonce":86634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":86634},{"type":"v1","addr":"172.21.15.50:6816","nonce":86634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":86634},{"type":"v1","addr":"172.21.15.50:6811","nonce":86634}]},"public_addr":"172.21.15.50:6801/86634","cluster_addr":"172.21.15.50:6805/86634","heartbeat_back_addr":"172.21.15.50:6816/86634","heartbeat_front_addr":"172.21.15.50:6811/86634","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.819964+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.795270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.734518+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.507975+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.783023+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T01:54:41.746894+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T01:59:55.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.456+0000 7f5396ada700 1 -- 172.21.15.7:0/1573998677 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5374035830 msgr2=0x7f5374037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:55.456 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.456+0000 7f5396ada700 1 --2- 172.21.15.7:0/1573998677 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5374035830 0x7f5374037ce0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5384011310 tx=0x7f53840091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:55.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.456+0000 7f5396ada700 10 monclient: shutdown 2023-09-30T01:59:55.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.456+0000 7f5396ada700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:55.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.456+0000 7f5396ada700 1 -- 172.21.15.7:0/1573998677 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f53900a2390 msgr2=0x7f539012e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:55.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.456+0000 7f5396ada700 1 --2- 172.21.15.7:0/1573998677 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f53900a2390 0x7f539012e100 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f538800cf60 tx=0x7f5388007840 comp rx=0 tx=0).stop 2023-09-30T01:59:55.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.456+0000 7f5396ada700 1 -- 172.21.15.7:0/1573998677 shutdown_connections 2023-09-30T01:59:55.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.456+0000 7f5396ada700 1 --2- 172.21.15.7:0/1573998677 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f5374035830 0x7f5374037ce0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:55.457 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.456+0000 7f5396ada700 1 --2- 172.21.15.7:0/1573998677 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5390096a50 0x7f53900a0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:55.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.456+0000 7f5396ada700 1 --2- 172.21.15.7:0/1573998677 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f53900a2390 0x7f539012e100 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:55.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.456+0000 7f5396ada700 1 --2- 172.21.15.7:0/1573998677 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5390097360 0x7f539011c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:55.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.456+0000 7f5396ada700 1 -- 172.21.15.7:0/1573998677 >> 172.21.15.7:0/1573998677 conn(0x7f5390070cf0 msgr2=0x7f53901307e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:55.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.456+0000 7f5396ada700 1 -- 172.21.15.7:0/1573998677 shutdown_connections 2023-09-30T01:59:55.458 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:55.456+0000 7f5396ada700 1 -- 172.21.15.7:0/1573998677 wait complete. 2023-09-30T01:59:56.469 INFO:tasks.ceph:Scrubbing osd.0 2023-09-30T01:59:56.469 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:59:56.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.605+0000 7f71fc55b700 10 monclient: get_monmap_and_config 2023-09-30T01:59:56.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.605+0000 7f71fc55b700 10 monclient: build_initial_monmap 2023-09-30T01:59:56.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.605+0000 7f71fc55b700 10 monclient: monmap: 2023-09-30T01:59:56.605 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:56.606136+0000 2023-09-30T01:59:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:56.606136+0000 2023-09-30T01:59:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:56.606 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:56.607 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:56.607 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:56.607 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:56.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.605+0000 7f71fc55b700 1 Processor -- start 2023-09-30T01:59:56.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.605+0000 7f71fc55b700 1 -- start start 2023-09-30T01:59:56.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.605+0000 7f71fc55b700 10 monclient: init 2023-09-30T01:59:56.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71fc55b700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:56.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71fc55b700 10 monclient: _add_conns ranks=[4,2,1,3,0] 2023-09-30T01:59:56.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71fc55b700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f71f40a2900 0x7f71f4096a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:56.617 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71fc55b700 10 monclient(hunting): picked mon.noname-c con 0x7f71f40a2900 addr v2:172.21.15.50:3300/0 2023-09-30T01:59:56.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71fc55b700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f71f4096f90 0x7f71f40973b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:56.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71fc55b700 10 monclient(hunting): picked mon.noname-b con 0x7f71f4096f90 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:56.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71fc55b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f71f41321b0 addr v1:172.21.15.50:6789/0 2023-09-30T01:59:56.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71fc55b700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f40991c0 con 0x7f71f41321b0 2023-09-30T01:59:56.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71fc55b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:56.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71fc55b700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f40978f0 con 0x7f71f40a2900 2023-09-30T01:59:56.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71fc55b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:56.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71fc55b700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f4098ea0 con 0x7f71f4096f90 2023-09-30T01:59:56.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71fc55b700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:56.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71fc55b700 10 monclient(hunting): authenticate will time out at 1485.671566s 2023-09-30T01:59:56.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71f9af6700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f71f4096f90 0x7f71f40973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:56.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71f9af6700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f71f4096f90 0x7f71f40973b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41338/0 (socket says 172.21.15.7:41338) 2023-09-30T01:59:56.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71f9af6700 1 -- 172.21.15.7:0/1156384797 learned_addr learned my addr 172.21.15.7:0/1156384797 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:56.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71f9af6700 10 monclient(hunting): get_auth_request con 0x7f71f4096f90 auth_method 0 2023-09-30T01:59:56.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71f9af6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:56.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71f9af6700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:56.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71f9af6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:56.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71f9af6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:56.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71f9af6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:56.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71f9af6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:56.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71fa2f7700 1 --2- 172.21.15.7:0/1156384797 >> v2:172.21.15.50:3300/0 conn(0x7f71f40a2900 0x7f71f4096a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:56.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71fa2f7700 10 monclient(hunting): get_auth_request con 0x7f71f40a2900 auth_method 0 2023-09-30T01:59:56.619 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71fa2f7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:56.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71fa2f7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:56.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71fa2f7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:56.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71f9af6700 10 monclient(hunting): handle_auth_done global_id 4778 payload 274 2023-09-30T01:59:56.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.606+0000 7f71f9af6700 1 -- 172.21.15.7:0/1156384797 >> v2:172.21.15.50:3300/0 conn(0x7f71f40a2900 msgr2=0x7f71f4096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:56.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.607+0000 7f71f9af6700 1 --2- 172.21.15.7:0/1156384797 >> v2:172.21.15.50:3300/0 conn(0x7f71f40a2900 0x7f71f4096a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:56.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.607+0000 7f71f9af6700 1 -- 172.21.15.7:0/1156384797 >> v1:172.21.15.50:6789/0 conn(0x7f71f41321b0 legacy=0x7f71f4132600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:56.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.607+0000 7f71f9af6700 10 monclient: _finish_hunting 0 2023-09-30T01:59:56.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.607+0000 7f71f9af6700 1 monclient: found mon.noname-b 2023-09-30T01:59:56.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.607+0000 7f71f9af6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:56.620 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.607+0000 7f71f9af6700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:56.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.607+0000 7f71f9af6700 1 -- 172.21.15.7:0/1156384797 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f40a16d0 con 0x7f71f4096f90 2023-09-30T01:59:56.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.607+0000 7f71fa2f7700 1 --2- 172.21.15.7:0/1156384797 >> v2:172.21.15.50:3300/0 conn(0x7f71f40a2900 0x7f71f4096a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T01:59:56.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71f9af6700 1 -- 172.21.15.7:0/1156384797 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f71f4096f90 msgr2=0x7f71f40973b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:56.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71f9af6700 1 --2- 172.21.15.7:0/1156384797 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f71f4096f90 0x7f71f40973b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f71e40099b0 tx=0x7f71e4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c56b8d771a909b2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:56.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71f92f5700 1 -- 172.21.15.7:0/1156384797 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e400ba30 con 0x7f71f4096f90 2023-09-30T01:59:56.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71f92f5700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:56.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71f92f5700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:56.621 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71f92f5700 10 monclient: dump: 2023-09-30T01:59:56.621 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:56.621 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:56.621 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:56.621 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:56.622 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:56.622 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:56.622 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:56.622 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:56.622 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:56.622 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:56.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71f92f5700 10 monclient: _finish_auth 0 2023-09-30T01:59:56.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71f92f5700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:56.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71f92f5700 1 -- 172.21.15.7:0/1156384797 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e400bb90 con 0x7f71f4096f90 2023-09-30T01:59:56.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71fc55b700 5 monclient: authenticate success, global_id 4778 2023-09-30T01:59:56.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71fc55b700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:56.622 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71f92f5700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:56.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71fc55b700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:56.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71fc55b700 4 set_mon_vals no callback set 2023-09-30T01:59:56.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71f92f5700 1 -- 172.21.15.7:0/1156384797 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e4011420 con 0x7f71f4096f90 2023-09-30T01:59:56.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71f92f5700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:56.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71f92f5700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:56.623 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71f92f5700 10 monclient: dump: 2023-09-30T01:59:56.623 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:56.623 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:56.624 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:56.624 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:56.624 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:56.624 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:56.624 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:56.624 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:56.624 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:56.624 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:56.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71fc55b700 10 monclient: shutdown 2023-09-30T01:59:56.624 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71fc55b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:56.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71fc55b700 1 -- 172.21.15.7:0/1156384797 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f71f4096f90 msgr2=0x7f71f40973b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:56.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.609+0000 7f71fc55b700 1 --2- 172.21.15.7:0/1156384797 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f71f4096f90 0x7f71f40973b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f71e40099b0 tx=0x7f71e4009250 comp rx=0 tx=0).stop 2023-09-30T01:59:56.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.610+0000 7f71fc55b700 1 -- 172.21.15.7:0/1156384797 shutdown_connections 2023-09-30T01:59:56.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.610+0000 7f71fc55b700 1 --2- 172.21.15.7:0/1156384797 >> v2:172.21.15.50:3300/0 conn(0x7f71f40a2900 0x7f71f4096a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:56.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.610+0000 7f71fc55b700 1 --2- 172.21.15.7:0/1156384797 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f71f4096f90 0x7f71f40973b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:56.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.610+0000 7f71fc55b700 1 -- 172.21.15.7:0/1156384797 >> 172.21.15.7:0/1156384797 conn(0x7f71f40a5cd0 msgr2=0x7f71f40a21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:56.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.610+0000 7f71fc55b700 1 -- 172.21.15.7:0/1156384797 shutdown_connections 2023-09-30T01:59:56.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.610+0000 7f71fc55b700 1 -- 172.21.15.7:0/1156384797 wait complete. 2023-09-30T01:59:56.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.610+0000 7f71fc55b700 10 monclient: build_initial_monmap 2023-09-30T01:59:56.625 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.610+0000 7f71fc55b700 10 monclient: monmap: 2023-09-30T01:59:56.626 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:56.626 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:56.626 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:56.626 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:56.626 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:56.626 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:56.627 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:56.627 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:56.627 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:56.627 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:56.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.610+0000 7f71fc55b700 1 Processor -- start 2023-09-30T01:59:56.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fc55b700 1 -- start start 2023-09-30T01:59:56.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fc55b700 10 monclient: init 2023-09-30T01:59:56.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fc55b700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:56.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fc55b700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T01:59:56.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fc55b700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f71f4096f90 0x7f71f412ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:56.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fc55b700 10 monclient(hunting): picked mon.noname-c con 0x7f71f4096f90 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:56.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fc55b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71f40a2900 0x7f71f409db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:56.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fc55b700 10 monclient(hunting): picked mon.noname-a con 0x7f71f40a2900 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:56.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fc55b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71f41321b0 0x7f71f409e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:56.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fc55b700 10 monclient(hunting): picked mon.noname-b con 0x7f71f41321b0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:56.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fc55b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:56.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fc55b700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f40a1280 con 0x7f71f40a2900 2023-09-30T01:59:56.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fc55b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:56.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fc55b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f4135620 con 0x7f71f41321b0 2023-09-30T01:59:56.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fc55b700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:56.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fc55b700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f411c860 con 0x7f71f4096f90 2023-09-30T01:59:56.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fc55b700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:56.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fc55b700 10 monclient(hunting): authenticate will time out at 1485.676452s 2023-09-30T01:59:56.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fa2f7700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f71f4096f90 0x7f71f412ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:56.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fa2f7700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f71f4096f90 0x7f71f412ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41352/0 (socket says 172.21.15.7:41352) 2023-09-30T01:59:56.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fa2f7700 1 -- 172.21.15.7:0/3403742607 learned_addr learned my addr 172.21.15.7:0/3403742607 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:56.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fa2f7700 10 monclient(hunting): get_auth_request con 0x7f71f4096f90 auth_method 0 2023-09-30T01:59:56.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fa2f7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:56.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fa2f7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:56.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fa2f7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:56.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71f9af6700 1 --2- 172.21.15.7:0/3403742607 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71f40a2900 0x7f71f409db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:56.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71f9af6700 10 monclient(hunting): get_auth_request con 0x7f71f40a2900 auth_method 0 2023-09-30T01:59:56.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71f9af6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:56.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71f9af6700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:56.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71f9af6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:56.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fa2f7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:56.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fa2f7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:56.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fa2f7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:56.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71faaf8700 1 --2- 172.21.15.7:0/3403742607 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71f41321b0 0x7f71f409e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:56.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71f9af6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:56.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71f9af6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:56.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71f9af6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:56.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71faaf8700 10 monclient(hunting): get_auth_request con 0x7f71f41321b0 auth_method 0 2023-09-30T01:59:56.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71faaf8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:56.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71faaf8700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:56.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71faaf8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:56.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71faaf8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:56.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71faaf8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:56.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71faaf8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:56.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fa2f7700 10 monclient(hunting): handle_auth_done global_id 4781 payload 824 2023-09-30T01:59:56.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fa2f7700 1 -- 172.21.15.7:0/3403742607 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71f41321b0 msgr2=0x7f71f409e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:56.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fa2f7700 1 --2- 172.21.15.7:0/3403742607 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71f41321b0 0x7f71f409e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:56.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fa2f7700 1 -- 172.21.15.7:0/3403742607 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71f40a2900 msgr2=0x7f71f409db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:56.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fa2f7700 1 --2- 172.21.15.7:0/3403742607 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71f40a2900 0x7f71f409db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:56.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fa2f7700 10 monclient: _finish_hunting 0 2023-09-30T01:59:56.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fa2f7700 1 monclient: found mon.noname-c 2023-09-30T01:59:56.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fa2f7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:56.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fa2f7700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:59:56.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71fa2f7700 1 -- 172.21.15.7:0/3403742607 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e40094f0 con 0x7f71f4096f90 2023-09-30T01:59:56.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.611+0000 7f71f9af6700 1 --2- 172.21.15.7:0/3403742607 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f71f40a2900 0x7f71f409db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:56.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71fa2f7700 1 -- 172.21.15.7:0/3403742607 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f71f4096f90 msgr2=0x7f71f412ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:56.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71fa2f7700 1 --2- 172.21.15.7:0/3403742607 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f71f4096f90 0x7f71f412ee20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f71f000e8d0 tx=0x7f71f0008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:56.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71ea7fc700 1 -- 172.21.15.7:0/3403742607 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f00093b0 con 0x7f71f4096f90 2023-09-30T01:59:56.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71ea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:56.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71ea7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:59:56.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71ea7fc700 10 monclient: dump: 2023-09-30T01:59:56.635 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:56.635 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:56.635 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:56.635 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:56.636 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:56.636 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:56.636 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:56.636 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:56.636 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:56.637 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:56.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71ea7fc700 10 monclient: _finish_auth 0 2023-09-30T01:59:56.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71ea7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:56.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71ea7fc700 1 -- 172.21.15.7:0/3403742607 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f0013620 con 0x7f71f4096f90 2023-09-30T01:59:56.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71fc55b700 5 monclient: authenticate success, global_id 4781 2023-09-30T01:59:56.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71ea7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:56.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71ea7fc700 1 -- 172.21.15.7:0/3403742607 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f00139c0 con 0x7f71f4096f90 2023-09-30T01:59:56.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71fc55b700 10 monclient: _renew_subs 2023-09-30T01:59:56.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71fc55b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:56.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71fc55b700 1 -- 172.21.15.7:0/3403742607 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f411bff0 con 0x7f71f4096f90 2023-09-30T01:59:56.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71f8af4700 4 set_mon_vals no callback set 2023-09-30T01:59:56.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71ea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:56.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71ea7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:56.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71ea7fc700 10 monclient: dump: 2023-09-30T01:59:56.640 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:56.640 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:56.640 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:56.640 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:56.640 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:56.641 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:56.641 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:56.641 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:56.641 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:56.641 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:56.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71fc55b700 10 monclient: _renew_subs 2023-09-30T01:59:56.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71fc55b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:56.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.614+0000 7f71fc55b700 1 -- 172.21.15.7:0/3403742607 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f409f780 con 0x7f71f4096f90 2023-09-30T01:59:56.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.615+0000 7f71fc55b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:56.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.615+0000 7f71fc55b700 1 -- 172.21.15.7:0/3403742607 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f71f411c390 con 0x7f71f4096f90 2023-09-30T01:59:56.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.615+0000 7f71ea7fc700 1 -- 172.21.15.7:0/3403742607 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f71f0012370 con 0x7f71f4096f90 2023-09-30T01:59:56.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.615+0000 7f71ea7fc700 1 --2- 172.21.15.7:0/3403742607 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f71d8035830 0x7f71d8037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:56.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.615+0000 7f71ea7fc700 1 -- 172.21.15.7:0/3403742607 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f71f0047970 con 0x7f71f4096f90 2023-09-30T01:59:56.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.615+0000 7f71ea7fc700 1 --2- 172.21.15.7:0/3403742607 >> [v2:172.21.15.7:6800/86601,v1:172.21.15.7:6801/86601] conn(0x7f71d803b780 0x7f71d803db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:56.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.615+0000 7f71ea7fc700 1 -- 172.21.15.7:0/3403742607 --> [v2:172.21.15.7:6800/86601,v1:172.21.15.7:6801/86601] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71d803e240 con 0x7f71d803b780 2023-09-30T01:59:56.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.615+0000 7f71ea7fc700 1 -- 172.21.15.7:0/3403742607 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71f004c2a0 con 0x7f71f4096f90 2023-09-30T01:59:56.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.615+0000 7f71ea7fc700 10 monclient: handle_get_version_reply finishing 1 version 15 2023-09-30T01:59:56.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.615+0000 7f71faaf8700 1 --2- 172.21.15.7:0/3403742607 >> [v2:172.21.15.7:6800/86601,v1:172.21.15.7:6801/86601] conn(0x7f71d803b780 0x7f71d803db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:56.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.615+0000 7f71faaf8700 10 monclient: get_auth_request con 0x7f71d803b780 auth_method 0 2023-09-30T01:59:56.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.615+0000 7f71f9af6700 1 --2- 172.21.15.7:0/3403742607 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f71d8035830 0x7f71d8037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:56.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.616+0000 7f71f9af6700 10 monclient: get_auth_request con 0x7f71d8035830 auth_method 0 2023-09-30T01:59:56.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.618+0000 7f71f9af6700 1 --2- 172.21.15.7:0/3403742607 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f71d8035830 0x7f71d8037ce0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f71e40099b0 tx=0x7f71e4016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:56.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.620+0000 7f71faaf8700 1 --2- 172.21.15.7:0/3403742607 >> [v2:172.21.15.7:6800/86601,v1:172.21.15.7:6801/86601] conn(0x7f71d803b780 0x7f71d803db90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:56.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.620+0000 7f71ea7fc700 1 -- 172.21.15.7:0/3403742607 <== osd.0 v2:172.21.15.7:6800/86601 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f71d803e240 con 0x7f71d803b780 2023-09-30T01:59:56.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.635+0000 7f71fc55b700 1 -- 172.21.15.7:0/3403742607 --> [v2:172.21.15.7:6800/86601,v1:172.21.15.7:6801/86601] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f71f4142890 con 0x7f71d803b780 2023-09-30T01:59:56.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.635+0000 7f71ea7fc700 1 -- 172.21.15.7:0/3403742607 <== osd.0 v2:172.21.15.7:6800/86601 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f71f4142890 con 0x7f71d803b780 2023-09-30T01:59:56.648 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-09-30T01:59:56.649 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:59:56.649 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-09-30T01:59:56.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.635+0000 7f71fc55b700 1 -- 172.21.15.7:0/3403742607 >> [v2:172.21.15.7:6800/86601,v1:172.21.15.7:6801/86601] conn(0x7f71d803b780 msgr2=0x7f71d803db90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:56.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.635+0000 7f71fc55b700 1 --2- 172.21.15.7:0/3403742607 >> [v2:172.21.15.7:6800/86601,v1:172.21.15.7:6801/86601] conn(0x7f71d803b780 0x7f71d803db90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:56.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.635+0000 7f71fc55b700 1 -- 172.21.15.7:0/3403742607 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f71d8035830 msgr2=0x7f71d8037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:56.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.635+0000 7f71fc55b700 1 --2- 172.21.15.7:0/3403742607 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f71d8035830 0x7f71d8037ce0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f71e40099b0 tx=0x7f71e4016000 comp rx=0 tx=0).stop 2023-09-30T01:59:56.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.635+0000 7f71fc55b700 10 monclient: shutdown 2023-09-30T01:59:56.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.635+0000 7f71fc55b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:56.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.635+0000 7f71fc55b700 1 -- 172.21.15.7:0/3403742607 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f71f4096f90 msgr2=0x7f71f412ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:56.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.635+0000 7f71fc55b700 1 --2- 172.21.15.7:0/3403742607 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f71f4096f90 0x7f71f412ee20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f71f000e8d0 tx=0x7f71f0008040 comp rx=0 tx=0).stop 2023-09-30T01:59:56.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.635+0000 7f71faaf8700 1 -- 172.21.15.7:0/3403742607 reap_dead start 2023-09-30T01:59:56.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.635+0000 7f71fc55b700 1 -- 172.21.15.7:0/3403742607 shutdown_connections 2023-09-30T01:59:56.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.635+0000 7f71fc55b700 1 -- 172.21.15.7:0/3403742607 >> 172.21.15.7:0/3403742607 conn(0x7f71f40a5cd0 msgr2=0x7f71f41347d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:56.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.635+0000 7f71fc55b700 1 -- 172.21.15.7:0/3403742607 shutdown_connections 2023-09-30T01:59:56.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.635+0000 7f71fc55b700 1 -- 172.21.15.7:0/3403742607 wait complete. 2023-09-30T01:59:56.653 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-09-30T01:59:56.771 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.771+0000 7fc1daef7700 10 monclient: get_monmap_and_config 2023-09-30T01:59:56.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.771+0000 7fc1daef7700 10 monclient: build_initial_monmap 2023-09-30T01:59:56.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.771+0000 7fc1daef7700 10 monclient: monmap: 2023-09-30T01:59:56.772 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:56.772 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:56.772 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:56.772395+0000 2023-09-30T01:59:56.772 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:56.772395+0000 2023-09-30T01:59:56.773 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:56.773 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:56.773 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:56.773 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:56.773 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:56.773 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:56.773 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:56.773 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:56.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1daef7700 1 Processor -- start 2023-09-30T01:59:56.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1daef7700 1 -- start start 2023-09-30T01:59:56.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1daef7700 10 monclient: init 2023-09-30T01:59:56.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1daef7700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:56.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1daef7700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2023-09-30T01:59:56.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1daef7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc1d4096a50 addr v1:172.21.15.50:6789/0 2023-09-30T01:59:56.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1daef7700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1d4096e20 0x7fc1d40971f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:56.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1daef7700 10 monclient(hunting): picked mon.noname-b con 0x7fc1d4096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:56.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1daef7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc1d4137710 addr v1:172.21.15.7:6789/0 2023-09-30T01:59:56.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1daef7700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d4132d90 con 0x7fc1d4137710 2023-09-30T01:59:56.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1daef7700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d4097b40 con 0x7fc1d4096a50 2023-09-30T01:59:56.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1daef7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:56.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1daef7700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d4099160 con 0x7fc1d4096e20 2023-09-30T01:59:56.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1daef7700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:56.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1daef7700 10 monclient(hunting): authenticate will time out at 1485.837724s 2023-09-30T01:59:56.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1d3fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1d4096e20 0x7fc1d40971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:56.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1d9494700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fc1d4137710 0x7fc1d4139af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48938/0 (socket says 172.21.15.7:48938) 2023-09-30T01:59:56.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1d3fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1d4096e20 0x7fc1d40971f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41354/0 (socket says 172.21.15.7:41354) 2023-09-30T01:59:56.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1d9494700 1 -- 172.21.15.7:0/631924207 learned_addr learned my addr 172.21.15.7:0/631924207 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:59:56.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1d3fff700 10 monclient(hunting): get_auth_request con 0x7fc1d4096e20 auth_method 0 2023-09-30T01:59:56.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1d3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:56.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1d3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:56.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1d3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:56.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1d3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:56.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1d3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:56.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.772+0000 7fc1d3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:56.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.773+0000 7fc1d3fff700 10 monclient(hunting): handle_auth_done global_id 4784 payload 274 2023-09-30T01:59:56.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.773+0000 7fc1d3fff700 1 -- 172.21.15.7:0/631924207 >> v1:172.21.15.50:6789/0 conn(0x7fc1d4096a50 legacy=0x7fc1d4132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:56.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.773+0000 7fc1d3fff700 1 -- 172.21.15.7:0/631924207 >> v1:172.21.15.7:6789/0 conn(0x7fc1d4137710 legacy=0x7fc1d4139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:56.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.773+0000 7fc1d3fff700 10 monclient: _finish_hunting 0 2023-09-30T01:59:56.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.773+0000 7fc1d3fff700 1 monclient: found mon.noname-b 2023-09-30T01:59:56.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.773+0000 7fc1d3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:56.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.773+0000 7fc1d3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:56.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.773+0000 7fc1d3fff700 1 -- 172.21.15.7:0/631924207 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d4099720 con 0x7fc1d4096e20 2023-09-30T01:59:56.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1d3fff700 1 -- 172.21.15.7:0/631924207 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1d4096e20 msgr2=0x7fc1d40971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:56.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1d3fff700 1 --2- 172.21.15.7:0/631924207 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1d4096e20 0x7fc1d40971f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc1c8009910 tx=0x7fc1c80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eef8778438fedf6a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:56.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1d37fe700 1 -- 172.21.15.7:0/631924207 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c800b9f0 con 0x7fc1d4096e20 2023-09-30T01:59:56.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1d37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:56.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1d37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:56.793 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1d37fe700 10 monclient: dump: 2023-09-30T01:59:56.793 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:56.793 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:56.793 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:56.793 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:56.793 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:56.793 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:56.793 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:56.793 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:56.794 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:56.794 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:56.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1d37fe700 10 monclient: _finish_auth 0 2023-09-30T01:59:56.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1d37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:56.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1d37fe700 1 -- 172.21.15.7:0/631924207 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c800bb50 con 0x7fc1d4096e20 2023-09-30T01:59:56.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1daef7700 5 monclient: authenticate success, global_id 4784 2023-09-30T01:59:56.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1daef7700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:56.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1d37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:56.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1daef7700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:56.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1daef7700 4 set_mon_vals no callback set 2023-09-30T01:59:56.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1d37fe700 1 -- 172.21.15.7:0/631924207 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c8011420 con 0x7fc1d4096e20 2023-09-30T01:59:56.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1d37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:56.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1d37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:56.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1d37fe700 10 monclient: dump: 2023-09-30T01:59:56.795 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:56.795 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:56.795 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:56.795 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:56.795 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:56.795 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:56.795 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:56.795 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:56.795 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:56.796 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:56.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1daef7700 10 monclient: shutdown 2023-09-30T01:59:56.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1daef7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:56.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1daef7700 1 -- 172.21.15.7:0/631924207 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1d4096e20 msgr2=0x7fc1d40971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:56.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.775+0000 7fc1daef7700 1 --2- 172.21.15.7:0/631924207 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1d4096e20 0x7fc1d40971f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc1c8009910 tx=0x7fc1c80091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:56.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.776+0000 7fc1daef7700 1 -- 172.21.15.7:0/631924207 shutdown_connections 2023-09-30T01:59:56.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.776+0000 7fc1daef7700 1 --2- 172.21.15.7:0/631924207 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1d4096e20 0x7fc1d40971f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:56.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.776+0000 7fc1daef7700 1 -- 172.21.15.7:0/631924207 >> 172.21.15.7:0/631924207 conn(0x7fc1d4070cf0 msgr2=0x7fc1d40710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:56.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.776+0000 7fc1daef7700 1 -- 172.21.15.7:0/631924207 shutdown_connections 2023-09-30T01:59:56.796 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.776+0000 7fc1daef7700 1 -- 172.21.15.7:0/631924207 wait complete. 2023-09-30T01:59:56.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.776+0000 7fc1daef7700 10 monclient: build_initial_monmap 2023-09-30T01:59:56.797 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.776+0000 7fc1daef7700 10 monclient: monmap: 2023-09-30T01:59:56.797 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:56.797 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:56.797 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:56.797 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:56.797 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:56.797 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:56.797 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:56.798 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:56.798 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:56.798 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:56.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.776+0000 7fc1daef7700 1 Processor -- start 2023-09-30T01:59:56.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1daef7700 1 -- start start 2023-09-30T01:59:56.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1daef7700 10 monclient: init 2023-09-30T01:59:56.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1daef7700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:56.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1daef7700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:59:56.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1daef7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1d4096a50 0x7fc1d40a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:56.798 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1daef7700 10 monclient(hunting): picked mon.noname-a con 0x7fc1d4096a50 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:56.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1daef7700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1d4096e20 0x7fc1d409ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:56.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1daef7700 10 monclient(hunting): picked mon.noname-c con 0x7fc1d4096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:56.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1daef7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc1d4137710 0x7fc1d409cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:56.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1daef7700 10 monclient(hunting): picked mon.noname-b con 0x7fc1d4137710 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:56.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1daef7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:56.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1daef7700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d4131de0 con 0x7fc1d4096a50 2023-09-30T01:59:56.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1daef7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:56.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1daef7700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d40a07b0 con 0x7fc1d4137710 2023-09-30T01:59:56.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1daef7700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:56.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1daef7700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d409e670 con 0x7fc1d4096e20 2023-09-30T01:59:56.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1daef7700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:56.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1daef7700 10 monclient(hunting): authenticate will time out at 1485.842525s 2023-09-30T01:59:56.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d3fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1d4096e20 0x7fc1d409ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:56.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d8c93700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1d4096a50 0x7fc1d40a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:56.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d3fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1d4096e20 0x7fc1d409ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41366/0 (socket says 172.21.15.7:41366) 2023-09-30T01:59:56.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d3fff700 1 -- 172.21.15.7:0/547714126 learned_addr learned my addr 172.21.15.7:0/547714126 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:56.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d8c93700 10 monclient(hunting): get_auth_request con 0x7fc1d4096a50 auth_method 0 2023-09-30T01:59:56.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d8c93700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:56.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d8c93700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:56.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d8c93700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:56.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d3fff700 10 monclient(hunting): get_auth_request con 0x7fc1d4096e20 auth_method 0 2023-09-30T01:59:56.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:56.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:56.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:56.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d8c93700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:56.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d8c93700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:56.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d8c93700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:56.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:56.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:56.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:56.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d9494700 1 --2- 172.21.15.7:0/547714126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc1d4137710 0x7fc1d409cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:56.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d9494700 10 monclient(hunting): get_auth_request con 0x7fc1d4137710 auth_method 0 2023-09-30T01:59:56.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d9494700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:56.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d9494700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:56.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d9494700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:56.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d3fff700 10 monclient(hunting): handle_auth_done global_id 4787 payload 824 2023-09-30T01:59:56.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d3fff700 1 -- 172.21.15.7:0/547714126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc1d4137710 msgr2=0x7fc1d409cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:56.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d3fff700 1 --2- 172.21.15.7:0/547714126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc1d4137710 0x7fc1d409cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:56.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d3fff700 1 -- 172.21.15.7:0/547714126 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1d4096a50 msgr2=0x7fc1d40a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:56.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d3fff700 1 --2- 172.21.15.7:0/547714126 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1d4096a50 0x7fc1d40a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:56.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d3fff700 10 monclient: _finish_hunting 0 2023-09-30T01:59:56.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d3fff700 1 monclient: found mon.noname-c 2023-09-30T01:59:56.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:56.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d3fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:59:56.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d3fff700 1 -- 172.21.15.7:0/547714126 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c8009420 con 0x7fc1d4096e20 2023-09-30T01:59:56.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.777+0000 7fc1d8c93700 1 --2- 172.21.15.7:0/547714126 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1d4096a50 0x7fc1d40a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:56.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1d3fff700 1 -- 172.21.15.7:0/547714126 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1d4096e20 msgr2=0x7fc1d409ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:56.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1d3fff700 1 --2- 172.21.15.7:0/547714126 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1d4096e20 0x7fc1d409ca90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc1c80030c0 tx=0x7fc1c8016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:56.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1d0ff9700 1 -- 172.21.15.7:0/547714126 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c800bed0 con 0x7fc1d4096e20 2023-09-30T01:59:56.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1d0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:56.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1d0ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:59:56.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1d0ff9700 10 monclient: dump: 2023-09-30T01:59:56.804 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:56.804 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:56.804 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:56.804 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:56.804 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:56.804 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:56.804 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:56.804 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:56.804 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:56.805 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:56.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1d0ff9700 10 monclient: _finish_auth 0 2023-09-30T01:59:56.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1d0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:56.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1d0ff9700 1 -- 172.21.15.7:0/547714126 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c8020070 con 0x7fc1d4096e20 2023-09-30T01:59:56.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1d0ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:56.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1d0ff9700 1 -- 172.21.15.7:0/547714126 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c801b6a0 con 0x7fc1d4096e20 2023-09-30T01:59:56.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1daef7700 5 monclient: authenticate success, global_id 4787 2023-09-30T01:59:56.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1daef7700 10 monclient: _renew_subs 2023-09-30T01:59:56.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1daef7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:56.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1daef7700 1 -- 172.21.15.7:0/547714126 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d409f260 con 0x7fc1d4096e20 2023-09-30T01:59:56.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1d2ffd700 4 set_mon_vals no callback set 2023-09-30T01:59:56.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1d0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:56.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1d0ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:56.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1d0ff9700 10 monclient: dump: 2023-09-30T01:59:56.806 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:56.806 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:56.806 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:56.806 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:56.806 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:56.806 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:56.806 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:56.806 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:56.807 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:56.807 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:56.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1daef7700 10 monclient: _renew_subs 2023-09-30T01:59:56.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1daef7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:56.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.780+0000 7fc1daef7700 1 -- 172.21.15.7:0/547714126 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d40a1350 con 0x7fc1d4096e20 2023-09-30T01:59:56.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.781+0000 7fc1d0ff9700 1 -- 172.21.15.7:0/547714126 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fc1c8022030 con 0x7fc1d4096e20 2023-09-30T01:59:56.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.781+0000 7fc1daef7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:56.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.781+0000 7fc1daef7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:56.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.781+0000 7fc1daef7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:56.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.781+0000 7fc1daef7700 1 -- 172.21.15.7:0/547714126 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d4064c10 con 0x7fc1d4096e20 2023-09-30T01:59:56.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.781+0000 7fc1d0ff9700 1 --2- 172.21.15.7:0/547714126 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc1c00357e0 0x7fc1c0037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:56.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.781+0000 7fc1d0ff9700 1 -- 172.21.15.7:0/547714126 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fc1c8049c00 con 0x7fc1d4096e20 2023-09-30T01:59:56.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.781+0000 7fc1d8c93700 1 --2- 172.21.15.7:0/547714126 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc1c00357e0 0x7fc1c0037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:56.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.781+0000 7fc1d8c93700 10 monclient: get_auth_request con 0x7fc1c00357e0 auth_method 0 2023-09-30T01:59:56.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.784+0000 7fc1d8c93700 1 --2- 172.21.15.7:0/547714126 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc1c00357e0 0x7fc1c0037c90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4002e00 tx=0x7fc1c4007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:56.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.786+0000 7fc1d0ff9700 1 -- 172.21.15.7:0/547714126 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc1c801c7d0 con 0x7fc1d4096e20 2023-09-30T01:59:56.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.786+0000 7fc1d0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:56.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.786+0000 7fc1d0ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:56.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.976+0000 7fc1daef7700 1 -- 172.21.15.7:0/547714126 --> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1d4064c10 con 0x7fc1c00357e0 2023-09-30T01:59:56.976 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.977+0000 7fc1d0ff9700 1 -- 172.21.15.7:0/547714126 <== mgr.4105 v2:172.21.15.50:6824/86200 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc1d4064c10 con 0x7fc1c00357e0 2023-09-30T01:59:56.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.978+0000 7fc1daef7700 1 -- 172.21.15.7:0/547714126 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc1c00357e0 msgr2=0x7fc1c0037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:56.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.978+0000 7fc1daef7700 1 --2- 172.21.15.7:0/547714126 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc1c00357e0 0x7fc1c0037c90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4002e00 tx=0x7fc1c4007ea0 comp rx=0 tx=0).stop 2023-09-30T01:59:56.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.978+0000 7fc1daef7700 10 monclient: shutdown 2023-09-30T01:59:56.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.978+0000 7fc1daef7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:56.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.979+0000 7fc1daef7700 1 -- 172.21.15.7:0/547714126 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1d4096e20 msgr2=0x7fc1d409ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:56.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.979+0000 7fc1daef7700 1 --2- 172.21.15.7:0/547714126 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1d4096e20 0x7fc1d409ca90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc1c80030c0 tx=0x7fc1c8016000 comp rx=0 tx=0).stop 2023-09-30T01:59:56.979 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.979+0000 7fc1daef7700 1 -- 172.21.15.7:0/547714126 shutdown_connections 2023-09-30T01:59:56.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.979+0000 7fc1daef7700 1 --2- 172.21.15.7:0/547714126 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fc1c00357e0 0x7fc1c0037c90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:56.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.979+0000 7fc1daef7700 1 --2- 172.21.15.7:0/547714126 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1d4096a50 0x7fc1d40a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:56.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.979+0000 7fc1daef7700 1 --2- 172.21.15.7:0/547714126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc1d4137710 0x7fc1d409cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:56.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.979+0000 7fc1daef7700 1 --2- 172.21.15.7:0/547714126 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fc1d4096e20 0x7fc1d409ca90 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:56.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.979+0000 7fc1daef7700 1 -- 172.21.15.7:0/547714126 >> 172.21.15.7:0/547714126 conn(0x7fc1d4070cf0 msgr2=0x7fc1d412f110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:56.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.979+0000 7fc1daef7700 1 -- 172.21.15.7:0/547714126 shutdown_connections 2023-09-30T01:59:56.980 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:56.979+0000 7fc1daef7700 1 -- 172.21.15.7:0/547714126 wait complete. 2023-09-30T01:59:56.980 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 0 to deep-scrub 2023-09-30T01:59:56.990 INFO:tasks.ceph:Scrubbing osd.1 2023-09-30T01:59:56.991 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:59:57.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.112+0000 7f79c54c8700 10 monclient: get_monmap_and_config 2023-09-30T01:59:57.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.112+0000 7f79c54c8700 10 monclient: build_initial_monmap 2023-09-30T01:59:57.112 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.112+0000 7f79c54c8700 10 monclient: monmap: 2023-09-30T01:59:57.112 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:57.112 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:57.112 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:57.113218+0000 2023-09-30T01:59:57.112 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:57.113218+0000 2023-09-30T01:59:57.113 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:57.113 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.113 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:57.113 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:57.113 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:57.113 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:57.113 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:57.113 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.117 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.112+0000 7f79c54c8700 1 Processor -- start 2023-09-30T01:59:57.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.112+0000 7f79c54c8700 1 -- start start 2023-09-30T01:59:57.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.112+0000 7f79c54c8700 10 monclient: init 2023-09-30T01:59:57.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79c54c8700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:57.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79c54c8700 10 monclient: _add_conns ranks=[4,0,3,2,1] 2023-09-30T01:59:57.118 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79c54c8700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f79c0132280 0x7f79c0132650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79c54c8700 10 monclient(hunting): picked mon.noname-c con 0x7f79c0132280 addr v2:172.21.15.50:3300/0 2023-09-30T01:59:57.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79c54c8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f79c00a2270 addr v1:172.21.15.7:6789/0 2023-09-30T01:59:57.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79c54c8700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f79c01373a0 0x7f79c0139780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79c54c8700 10 monclient(hunting): picked mon.noname-a con 0x7f79c01373a0 addr v2:172.21.15.7:3300/0 2023-09-30T01:59:57.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79c54c8700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c009a880 con 0x7f79c00a2270 2023-09-30T01:59:57.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79c54c8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:57.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79c54c8700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79c009a510 con 0x7f79c01373a0 2023-09-30T01:59:57.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79c54c8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:57.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79c54c8700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79c00996c0 con 0x7f79c0132280 2023-09-30T01:59:57.119 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79c54c8700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:57.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79c54c8700 10 monclient(hunting): authenticate will time out at 1486.178276s 2023-09-30T01:59:57.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79bf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f79c01373a0 0x7f79c0139780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79bf7fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f79c01373a0 0x7f79c0139780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49342/0 (socket says 172.21.15.7:49342) 2023-09-30T01:59:57.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79bf7fe700 1 -- 172.21.15.7:0/3075554305 learned_addr learned my addr 172.21.15.7:0/3075554305 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:57.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79bf7fe700 10 monclient(hunting): get_auth_request con 0x7f79c01373a0 auth_method 0 2023-09-30T01:59:57.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79bf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:57.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79bf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:57.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79bf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:57.120 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79bf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:57.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79bf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:57.121 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79bf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:57.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79beffd700 1 --2- 172.21.15.7:0/3075554305 >> v2:172.21.15.50:3300/0 conn(0x7f79c0132280 0x7f79c0132650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79beffd700 10 monclient(hunting): get_auth_request con 0x7f79c0132280 auth_method 0 2023-09-30T01:59:57.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79beffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:57.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79beffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:57.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79beffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:57.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79beffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:57.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79beffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:57.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79beffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:57.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79beffd700 10 monclient(hunting): handle_auth_done global_id 4633 payload 274 2023-09-30T01:59:57.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79beffd700 1 -- 172.21.15.7:0/3075554305 >> v2:172.21.15.7:3300/0 conn(0x7f79c01373a0 msgr2=0x7f79c0139780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:57.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79beffd700 1 --2- 172.21.15.7:0/3075554305 >> v2:172.21.15.7:3300/0 conn(0x7f79c01373a0 0x7f79c0139780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79beffd700 1 -- 172.21.15.7:0/3075554305 >> v1:172.21.15.7:6789/0 conn(0x7f79c00a2270 legacy=0x7f79c00a2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:57.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79beffd700 10 monclient: _finish_hunting 0 2023-09-30T01:59:57.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79beffd700 1 monclient: found mon.noname-c 2023-09-30T01:59:57.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79beffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:57.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79beffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3300/0 2023-09-30T01:59:57.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.113+0000 7f79beffd700 1 -- 172.21.15.7:0/3075554305 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c00a2d80 con 0x7f79c0132280 2023-09-30T01:59:57.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79beffd700 1 -- 172.21.15.7:0/3075554305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79c0132280 msgr2=0x7f79c0132650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:57.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79beffd700 1 --2- 172.21.15.7:0/3075554305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79c0132280 0x7f79c0132650 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f79c0097040 tx=0x7f79b0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80b7ac71b2583ce2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:57.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79bdffb700 1 -- 172.21.15.7:0/3075554305 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b000e070 con 0x7f79c0132280 2023-09-30T01:59:57.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79bdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:57.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79bdffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:59:57.124 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79bdffb700 10 monclient: dump: 2023-09-30T01:59:57.124 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:57.124 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:57.124 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.124 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.124 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:57.124 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.124 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:57.124 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:57.125 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:57.125 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79bdffb700 10 monclient: _finish_auth 0 2023-09-30T01:59:57.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79bdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:57.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79bdffb700 1 -- 172.21.15.7:0/3075554305 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b000bd30 con 0x7f79c0132280 2023-09-30T01:59:57.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79c54c8700 5 monclient: authenticate success, global_id 4633 2023-09-30T01:59:57.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79c54c8700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:57.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79bdffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:57.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79c54c8700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:57.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79bdffb700 1 -- 172.21.15.7:0/3075554305 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b000bea0 con 0x7f79c0132280 2023-09-30T01:59:57.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79c54c8700 4 set_mon_vals no callback set 2023-09-30T01:59:57.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79bdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:57.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79bdffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:59:57.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79bdffb700 10 monclient: dump: 2023-09-30T01:59:57.126 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:57.126 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:57.126 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.126 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.126 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:57.126 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.126 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:57.127 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:57.127 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:57.127 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79c54c8700 10 monclient: shutdown 2023-09-30T01:59:57.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79c54c8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:57.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79c54c8700 1 -- 172.21.15.7:0/3075554305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79c0132280 msgr2=0x7f79c0132650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.116+0000 7f79c54c8700 1 --2- 172.21.15.7:0/3075554305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79c0132280 0x7f79c0132650 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f79c0097040 tx=0x7f79b0009780 comp rx=0 tx=0).stop 2023-09-30T01:59:57.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.117+0000 7f79c54c8700 1 -- 172.21.15.7:0/3075554305 shutdown_connections 2023-09-30T01:59:57.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.117+0000 7f79c54c8700 1 --2- 172.21.15.7:0/3075554305 >> v2:172.21.15.7:3300/0 conn(0x7f79c01373a0 0x7f79c0139780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.117+0000 7f79c54c8700 1 --2- 172.21.15.7:0/3075554305 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79c0132280 0x7f79c0132650 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.117+0000 7f79c54c8700 1 -- 172.21.15.7:0/3075554305 >> 172.21.15.7:0/3075554305 conn(0x7f79c00677e0 msgr2=0x7f79c0067be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:57.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.117+0000 7f79c54c8700 1 -- 172.21.15.7:0/3075554305 shutdown_connections 2023-09-30T01:59:57.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.117+0000 7f79c54c8700 1 -- 172.21.15.7:0/3075554305 wait complete. 2023-09-30T01:59:57.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.117+0000 7f79c54c8700 10 monclient: build_initial_monmap 2023-09-30T01:59:57.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.117+0000 7f79c54c8700 10 monclient: monmap: 2023-09-30T01:59:57.128 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:57.128 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:57.128 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:57.128 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:57.128 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:57.129 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.129 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:57.129 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:57.129 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:57.129 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.117+0000 7f79c54c8700 1 Processor -- start 2023-09-30T01:59:57.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79c54c8700 1 -- start start 2023-09-30T01:59:57.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79c54c8700 10 monclient: init 2023-09-30T01:59:57.135 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79c54c8700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:57.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79c54c8700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:59:57.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79c54c8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79c00a2270 0x7f79c01420b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79c54c8700 10 monclient(hunting): picked mon.noname-b con 0x7f79c00a2270 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:57.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79c54c8700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79c0132280 0x7f79c01425f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79c54c8700 10 monclient(hunting): picked mon.noname-c con 0x7f79c0132280 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:57.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79c54c8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79c01373a0 0x7f79c009daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79c54c8700 10 monclient(hunting): picked mon.noname-a con 0x7f79c01373a0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:57.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79c54c8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:57.136 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79c54c8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c00a2080 con 0x7f79c01373a0 2023-09-30T01:59:57.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79c54c8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:57.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79c54c8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c009fc80 con 0x7f79c00a2270 2023-09-30T01:59:57.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79c54c8700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:57.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79c54c8700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c011c130 con 0x7f79c0132280 2023-09-30T01:59:57.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79c54c8700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:57.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79c54c8700 10 monclient(hunting): authenticate will time out at 1486.183495s 2023-09-30T01:59:57.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79bf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79c01373a0 0x7f79c009daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79bf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79c01373a0 0x7f79c009daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49358/0 (socket says 172.21.15.7:49358) 2023-09-30T01:59:57.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79bf7fe700 1 -- 172.21.15.7:0/2199484029 learned_addr learned my addr 172.21.15.7:0/2199484029 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:57.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79bf7fe700 10 monclient(hunting): get_auth_request con 0x7f79c01373a0 auth_method 0 2023-09-30T01:59:57.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79bf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:57.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79bf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:57.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79be7fc700 1 --2- 172.21.15.7:0/2199484029 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79c0132280 0x7f79c01425f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79bf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:57.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79be7fc700 10 monclient(hunting): get_auth_request con 0x7f79c0132280 auth_method 0 2023-09-30T01:59:57.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79be7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:57.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79be7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:57.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79be7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:57.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79bf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:57.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79bf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:57.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79bf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:57.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79be7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:57.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79be7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:57.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79be7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:57.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79beffd700 1 --2- 172.21.15.7:0/2199484029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79c00a2270 0x7f79c01420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79bf7fe700 10 monclient(hunting): handle_auth_done global_id 4761 payload 824 2023-09-30T01:59:57.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79bf7fe700 1 -- 172.21.15.7:0/2199484029 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79c0132280 msgr2=0x7f79c01425f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79bf7fe700 1 --2- 172.21.15.7:0/2199484029 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f79c0132280 0x7f79c01425f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79bf7fe700 1 -- 172.21.15.7:0/2199484029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79c00a2270 msgr2=0x7f79c01420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79bf7fe700 1 --2- 172.21.15.7:0/2199484029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79c00a2270 0x7f79c01420b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79bf7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:59:57.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79bf7fe700 1 monclient: found mon.noname-a 2023-09-30T01:59:57.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79bf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:57.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79bf7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:57.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79bf7fe700 1 -- 172.21.15.7:0/2199484029 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b0009a20 con 0x7f79c01373a0 2023-09-30T01:59:57.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79beffd700 10 monclient: get_auth_request con 0x7f79c00a2270 auth_method 0 2023-09-30T01:59:57.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.118+0000 7f79beffd700 1 --2- 172.21.15.7:0/2199484029 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79c00a2270 0x7f79c01420b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:59:57.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79bf7fe700 1 -- 172.21.15.7:0/2199484029 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79c01373a0 msgr2=0x7f79c009daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:57.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79bf7fe700 1 --2- 172.21.15.7:0/2199484029 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79c01373a0 0x7f79c009daf0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f79b800e930 tx=0x7f79b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:57.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79b6ffd700 1 -- 172.21.15.7:0/2199484029 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b8013890 con 0x7f79c01373a0 2023-09-30T01:59:57.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79b6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:57.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79b6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:57.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79b6ffd700 10 monclient: dump: 2023-09-30T01:59:57.141 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:57.141 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:57.142 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.142 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.142 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:57.142 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.142 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:57.142 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:57.142 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:57.142 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79b6ffd700 10 monclient: _finish_auth 0 2023-09-30T01:59:57.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79b6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:57.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79c54c8700 5 monclient: authenticate success, global_id 4761 2023-09-30T01:59:57.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79b6ffd700 1 -- 172.21.15.7:0/2199484029 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b80139f0 con 0x7f79c01373a0 2023-09-30T01:59:57.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79b6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:57.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79c54c8700 10 monclient: _renew_subs 2023-09-30T01:59:57.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79b6ffd700 1 -- 172.21.15.7:0/2199484029 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b8013db0 con 0x7f79c01373a0 2023-09-30T01:59:57.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79c54c8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:57.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79c54c8700 1 -- 172.21.15.7:0/2199484029 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79c00a11c0 con 0x7f79c01373a0 2023-09-30T01:59:57.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79bd7fa700 4 set_mon_vals no callback set 2023-09-30T01:59:57.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79b6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:57.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79b6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:57.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79b6ffd700 10 monclient: dump: 2023-09-30T01:59:57.143 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:57.143 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:57.144 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.144 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.144 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:57.144 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.144 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:57.144 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:57.144 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:57.144 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79c54c8700 10 monclient: _renew_subs 2023-09-30T01:59:57.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79c54c8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:57.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.121+0000 7f79c54c8700 1 -- 172.21.15.7:0/2199484029 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79c009f690 con 0x7f79c01373a0 2023-09-30T01:59:57.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.122+0000 7f79c54c8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:57.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.122+0000 7f79c54c8700 1 -- 172.21.15.7:0/2199484029 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f79c00a2640 con 0x7f79c01373a0 2023-09-30T01:59:57.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.122+0000 7f79b6ffd700 1 -- 172.21.15.7:0/2199484029 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f79b800c670 con 0x7f79c01373a0 2023-09-30T01:59:57.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.122+0000 7f79b6ffd700 1 --2- 172.21.15.7:0/2199484029 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f79a00357e0 0x7f79a0037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.122+0000 7f79b6ffd700 1 -- 172.21.15.7:0/2199484029 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f79b8047ee0 con 0x7f79c01373a0 2023-09-30T01:59:57.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.122+0000 7f79b6ffd700 1 --2- 172.21.15.7:0/2199484029 >> [v2:172.21.15.7:6816/86600,v1:172.21.15.7:6817/86600] conn(0x7f79a003b660 0x7f79a003da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.122+0000 7f79b6ffd700 1 -- 172.21.15.7:0/2199484029 --> [v2:172.21.15.7:6816/86600,v1:172.21.15.7:6817/86600] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f79a003e120 con 0x7f79a003b660 2023-09-30T01:59:57.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.122+0000 7f79b6ffd700 1 -- 172.21.15.7:0/2199484029 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79b804a470 con 0x7f79c01373a0 2023-09-30T01:59:57.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.122+0000 7f79b6ffd700 10 monclient: handle_get_version_reply finishing 1 version 15 2023-09-30T01:59:57.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.122+0000 7f79be7fc700 1 --2- 172.21.15.7:0/2199484029 >> [v2:172.21.15.7:6816/86600,v1:172.21.15.7:6817/86600] conn(0x7f79a003b660 0x7f79a003da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.122+0000 7f79be7fc700 10 monclient: get_auth_request con 0x7f79a003b660 auth_method 0 2023-09-30T01:59:57.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.122+0000 7f79beffd700 1 --2- 172.21.15.7:0/2199484029 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f79a00357e0 0x7f79a0037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.122+0000 7f79beffd700 10 monclient: get_auth_request con 0x7f79a00357e0 auth_method 0 2023-09-30T01:59:57.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.125+0000 7f79beffd700 1 --2- 172.21.15.7:0/2199484029 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f79a00357e0 0x7f79a0037c90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f79b0009fd0 tx=0x7f79b0018000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:57.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.127+0000 7f79be7fc700 1 --2- 172.21.15.7:0/2199484029 >> [v2:172.21.15.7:6816/86600,v1:172.21.15.7:6817/86600] conn(0x7f79a003b660 0x7f79a003da70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:57.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.127+0000 7f79b6ffd700 1 -- 172.21.15.7:0/2199484029 <== osd.1 v2:172.21.15.7:6816/86600 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f79a003e120 con 0x7f79a003b660 2023-09-30T01:59:57.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.141+0000 7f79c54c8700 1 -- 172.21.15.7:0/2199484029 --> [v2:172.21.15.7:6816/86600,v1:172.21.15.7:6817/86600] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f79c0064c10 con 0x7f79a003b660 2023-09-30T01:59:57.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.141+0000 7f79b6ffd700 1 -- 172.21.15.7:0/2199484029 <== osd.1 v2:172.21.15.7:6816/86600 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f79c0064c10 con 0x7f79a003b660 2023-09-30T01:59:57.147 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-09-30T01:59:57.147 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:59:57.147 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-09-30T01:59:57.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.142+0000 7f79c54c8700 1 -- 172.21.15.7:0/2199484029 >> [v2:172.21.15.7:6816/86600,v1:172.21.15.7:6817/86600] conn(0x7f79a003b660 msgr2=0x7f79a003da70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.142+0000 7f79c54c8700 1 --2- 172.21.15.7:0/2199484029 >> [v2:172.21.15.7:6816/86600,v1:172.21.15.7:6817/86600] conn(0x7f79a003b660 0x7f79a003da70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.142+0000 7f79c54c8700 1 -- 172.21.15.7:0/2199484029 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f79a00357e0 msgr2=0x7f79a0037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.142+0000 7f79c54c8700 1 --2- 172.21.15.7:0/2199484029 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f79a00357e0 0x7f79a0037c90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f79b0009fd0 tx=0x7f79b0018000 comp rx=0 tx=0).stop 2023-09-30T01:59:57.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.142+0000 7f79c54c8700 10 monclient: shutdown 2023-09-30T01:59:57.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.142+0000 7f79c54c8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:57.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.142+0000 7f79c54c8700 1 -- 172.21.15.7:0/2199484029 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79c01373a0 msgr2=0x7f79c009daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.142+0000 7f79c54c8700 1 --2- 172.21.15.7:0/2199484029 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f79c01373a0 0x7f79c009daf0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f79b800e930 tx=0x7f79b8009250 comp rx=0 tx=0).stop 2023-09-30T01:59:57.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.142+0000 7f79bf7fe700 1 -- 172.21.15.7:0/2199484029 reap_dead start 2023-09-30T01:59:57.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.142+0000 7f79c54c8700 1 -- 172.21.15.7:0/2199484029 shutdown_connections 2023-09-30T01:59:57.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.142+0000 7f79c54c8700 1 -- 172.21.15.7:0/2199484029 >> 172.21.15.7:0/2199484029 conn(0x7f79c00677e0 msgr2=0x7f79c01384f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:57.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.142+0000 7f79c54c8700 1 -- 172.21.15.7:0/2199484029 shutdown_connections 2023-09-30T01:59:57.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.142+0000 7f79c54c8700 1 -- 172.21.15.7:0/2199484029 wait complete. 2023-09-30T01:59:57.153 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-09-30T01:59:57.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.273+0000 7f066453a700 10 monclient: get_monmap_and_config 2023-09-30T01:59:57.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.273+0000 7f066453a700 10 monclient: build_initial_monmap 2023-09-30T01:59:57.273 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.273+0000 7f066453a700 10 monclient: monmap: 2023-09-30T01:59:57.273 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:57.274 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:57.274 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:57.274198+0000 2023-09-30T01:59:57.274 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:57.274198+0000 2023-09-30T01:59:57.274 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:57.274 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.274 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:57.274 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:57.274 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:57.274 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:57.274 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:57.274 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.274 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.273+0000 7f066453a700 1 Processor -- start 2023-09-30T01:59:57.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.273+0000 7f066453a700 1 -- start start 2023-09-30T01:59:57.275 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.273+0000 7f066453a700 10 monclient: init 2023-09-30T01:59:57.280 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f066453a700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:57.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f066453a700 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-09-30T01:59:57.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f066453a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f065c096a50 addr v1:172.21.15.7:6789/0 2023-09-30T01:59:57.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f066453a700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f065c096e20 0x7f065c0971f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f066453a700 10 monclient(hunting): picked mon.noname-b con 0x7f065c096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:57.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f066453a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f065c137710 0x7f065c139af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f066453a700 10 monclient(hunting): picked mon.noname-a con 0x7f065c137710 addr v2:172.21.15.7:3300/0 2023-09-30T01:59:57.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f066453a700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065c132d90 con 0x7f065c096a50 2023-09-30T01:59:57.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f066453a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:57.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f066453a700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f065c097b40 con 0x7f065c137710 2023-09-30T01:59:57.281 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f066453a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:57.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f066453a700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065c099160 con 0x7f065c096e20 2023-09-30T01:59:57.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f066453a700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:57.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f066453a700 10 monclient(hunting): authenticate will time out at 1486.339399s 2023-09-30T01:59:57.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0661ad5700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f065c096e20 0x7f065c0971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0662ad7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f065c137710 0x7f065c139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0661ad5700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f065c096e20 0x7f065c0971f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41378/0 (socket says 172.21.15.7:41378) 2023-09-30T01:59:57.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0662ad7700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f065c137710 0x7f065c139af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49364/0 (socket says 172.21.15.7:49364) 2023-09-30T01:59:57.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f06622d6700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f065c096a50 0x7f065c132680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48956/0 (socket says 172.21.15.7:48956) 2023-09-30T01:59:57.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0661ad5700 1 -- 172.21.15.7:0/1381210197 learned_addr learned my addr 172.21.15.7:0/1381210197 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:57.282 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0661ad5700 10 monclient(hunting): get_auth_request con 0x7f065c096e20 auth_method 0 2023-09-30T01:59:57.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0661ad5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:57.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0661ad5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:57.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0661ad5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:57.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0662ad7700 10 monclient(hunting): get_auth_request con 0x7f065c137710 auth_method 0 2023-09-30T01:59:57.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0662ad7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:57.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0662ad7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:57.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0662ad7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:57.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0661ad5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:57.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0661ad5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:57.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0661ad5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:57.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0661ad5700 10 monclient(hunting): handle_auth_done global_id 4793 payload 274 2023-09-30T01:59:57.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0661ad5700 1 -- 172.21.15.7:0/1381210197 >> v2:172.21.15.7:3300/0 conn(0x7f065c137710 msgr2=0x7f065c139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:57.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0661ad5700 1 --2- 172.21.15.7:0/1381210197 >> v2:172.21.15.7:3300/0 conn(0x7f065c137710 0x7f065c139af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0661ad5700 1 -- 172.21.15.7:0/1381210197 >> v1:172.21.15.7:6789/0 conn(0x7f065c096a50 legacy=0x7f065c132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:57.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0661ad5700 10 monclient: _finish_hunting 0 2023-09-30T01:59:57.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0661ad5700 1 monclient: found mon.noname-b 2023-09-30T01:59:57.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0661ad5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:57.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0661ad5700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:57.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.274+0000 7f0661ad5700 1 -- 172.21.15.7:0/1381210197 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065c099720 con 0x7f065c096e20 2023-09-30T01:59:57.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f0661ad5700 1 -- 172.21.15.7:0/1381210197 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f065c096e20 msgr2=0x7f065c0971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:57.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f0661ad5700 1 --2- 172.21.15.7:0/1381210197 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f065c096e20 0x7f065c0971f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f06500099b0 tx=0x7f0650009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ff219567ef3ef1b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:57.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f06612d4700 1 -- 172.21.15.7:0/1381210197 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065000ba30 con 0x7f065c096e20 2023-09-30T01:59:57.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f06612d4700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:57.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f06612d4700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:57.285 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f06612d4700 10 monclient: dump: 2023-09-30T01:59:57.285 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:57.285 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:57.285 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.285 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.285 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:57.285 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.285 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:57.285 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:57.285 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:57.285 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f06612d4700 10 monclient: _finish_auth 0 2023-09-30T01:59:57.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f06612d4700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:57.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f06612d4700 1 -- 172.21.15.7:0/1381210197 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f065000bb90 con 0x7f065c096e20 2023-09-30T01:59:57.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f066453a700 5 monclient: authenticate success, global_id 4793 2023-09-30T01:59:57.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f066453a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:57.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f06612d4700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:57.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f066453a700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:57.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f06612d4700 1 -- 172.21.15.7:0/1381210197 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0650011420 con 0x7f065c096e20 2023-09-30T01:59:57.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f06612d4700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:57.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f066453a700 4 set_mon_vals no callback set 2023-09-30T01:59:57.286 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f06612d4700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:57.287 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f06612d4700 10 monclient: dump: 2023-09-30T01:59:57.287 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:57.287 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:57.287 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.287 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.287 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:57.287 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.287 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:57.287 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:57.287 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:57.288 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f066453a700 10 monclient: shutdown 2023-09-30T01:59:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f066453a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f066453a700 1 -- 172.21.15.7:0/1381210197 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f065c096e20 msgr2=0x7f065c0971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f066453a700 1 --2- 172.21.15.7:0/1381210197 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f065c096e20 0x7f065c0971f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f06500099b0 tx=0x7f0650009250 comp rx=0 tx=0).stop 2023-09-30T01:59:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f066453a700 1 -- 172.21.15.7:0/1381210197 shutdown_connections 2023-09-30T01:59:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f066453a700 1 --2- 172.21.15.7:0/1381210197 >> v2:172.21.15.7:3300/0 conn(0x7f065c137710 0x7f065c139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f066453a700 1 --2- 172.21.15.7:0/1381210197 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f065c096e20 0x7f065c0971f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f066453a700 1 -- 172.21.15.7:0/1381210197 >> 172.21.15.7:0/1381210197 conn(0x7f065c070cf0 msgr2=0x7f065c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f066453a700 1 -- 172.21.15.7:0/1381210197 shutdown_connections 2023-09-30T01:59:57.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.277+0000 7f066453a700 1 -- 172.21.15.7:0/1381210197 wait complete. 2023-09-30T01:59:57.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.278+0000 7f066453a700 10 monclient: build_initial_monmap 2023-09-30T01:59:57.289 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.278+0000 7f066453a700 10 monclient: monmap: 2023-09-30T01:59:57.289 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:57.289 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:57.289 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:57.289 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:57.289 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:57.289 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.289 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:57.290 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:57.290 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:57.290 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.278+0000 7f066453a700 1 Processor -- start 2023-09-30T01:59:57.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.278+0000 7f066453a700 1 -- start start 2023-09-30T01:59:57.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.278+0000 7f066453a700 10 monclient: init 2023-09-30T01:59:57.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f066453a700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:57.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f066453a700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:59:57.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f066453a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f065c096a50 0x7f065c141dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.290 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f066453a700 10 monclient(hunting): picked mon.noname-b con 0x7f065c096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f066453a700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f065c096e20 0x7f065c142310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f066453a700 10 monclient(hunting): picked mon.noname-c con 0x7f065c096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f066453a700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f065c137710 0x7f065c09e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f066453a700 10 monclient(hunting): picked mon.noname-a con 0x7f065c137710 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f066453a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f066453a700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c11be70 con 0x7f065c137710 2023-09-30T01:59:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f066453a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f066453a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c0a1dc0 con 0x7f065c096a50 2023-09-30T01:59:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f066453a700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f066453a700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065c0a07f0 con 0x7f065c096e20 2023-09-30T01:59:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f066453a700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f066453a700 10 monclient(hunting): authenticate will time out at 1486.344082s 2023-09-30T01:59:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0662ad7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f065c137710 0x7f065c09e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0661ad5700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f065c096e20 0x7f065c142310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0662ad7700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f065c137710 0x7f065c09e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49366/0 (socket says 172.21.15.7:49366) 2023-09-30T01:59:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0661ad5700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f065c096e20 0x7f065c142310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41390/0 (socket says 172.21.15.7:41390) 2023-09-30T01:59:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0662ad7700 1 -- 172.21.15.7:0/1149570641 learned_addr learned my addr 172.21.15.7:0/1149570641 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0662ad7700 10 monclient(hunting): get_auth_request con 0x7f065c137710 auth_method 0 2023-09-30T01:59:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0662ad7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0662ad7700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0662ad7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:57.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0661ad5700 10 monclient(hunting): get_auth_request con 0x7f065c096e20 auth_method 0 2023-09-30T01:59:57.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0661ad5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:57.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0661ad5700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:57.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0661ad5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:57.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0662ad7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:57.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0662ad7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:57.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0662ad7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:57.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0661ad5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:57.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0661ad5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0661ad5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f06622d6700 1 --2- 172.21.15.7:0/1149570641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f065c096a50 0x7f065c141dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f06622d6700 10 monclient(hunting): get_auth_request con 0x7f065c096a50 auth_method 0 2023-09-30T01:59:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f06622d6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f06622d6700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f06622d6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0662ad7700 10 monclient(hunting): handle_auth_done global_id 4767 payload 824 2023-09-30T01:59:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0662ad7700 1 -- 172.21.15.7:0/1149570641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f065c096e20 msgr2=0x7f065c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0662ad7700 1 --2- 172.21.15.7:0/1149570641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f065c096e20 0x7f065c142310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0662ad7700 1 -- 172.21.15.7:0/1149570641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f065c096a50 msgr2=0x7f065c141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0662ad7700 1 --2- 172.21.15.7:0/1149570641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f065c096a50 0x7f065c141dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0662ad7700 10 monclient: _finish_hunting 0 2023-09-30T01:59:57.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0662ad7700 1 monclient: found mon.noname-a 2023-09-30T01:59:57.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0662ad7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:57.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0662ad7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:57.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0662ad7700 1 -- 172.21.15.7:0/1149570641 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06500094f0 con 0x7f065c137710 2023-09-30T01:59:57.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.279+0000 7f0661ad5700 1 --2- 172.21.15.7:0/1149570641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f065c096e20 0x7f065c142310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:57.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f0662ad7700 1 -- 172.21.15.7:0/1149570641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f065c137710 msgr2=0x7f065c09e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:57.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f0662ad7700 1 --2- 172.21.15.7:0/1149570641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f065c137710 0x7f065c09e6f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f065400d890 tx=0x7f0654008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:57.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f065a7fc700 1 -- 172.21.15.7:0/1149570641 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0654012800 con 0x7f065c137710 2023-09-30T01:59:57.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f065a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:57.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f065a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:57.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f065a7fc700 10 monclient: dump: 2023-09-30T01:59:57.298 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:57.298 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:57.298 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.298 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.298 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:57.298 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.298 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:57.298 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:57.298 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:57.299 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f065a7fc700 10 monclient: _finish_auth 0 2023-09-30T01:59:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f065a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f065a7fc700 1 -- 172.21.15.7:0/1149570641 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0654012980 con 0x7f065c137710 2023-09-30T01:59:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f066453a700 5 monclient: authenticate success, global_id 4767 2023-09-30T01:59:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f066453a700 10 monclient: _renew_subs 2023-09-30T01:59:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f066453a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f066453a700 1 -- 172.21.15.7:0/1149570641 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f065c131d10 con 0x7f065c137710 2023-09-30T01:59:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f065a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f065a7fc700 1 -- 172.21.15.7:0/1149570641 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0654012d40 con 0x7f065c137710 2023-09-30T01:59:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f065a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f065a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f065a7fc700 10 monclient: dump: 2023-09-30T01:59:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:57.301 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:57.301 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:57.301 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f066453a700 10 monclient: _renew_subs 2023-09-30T01:59:57.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f066453a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:57.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f0660ad3700 4 set_mon_vals no callback set 2023-09-30T01:59:57.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.281+0000 7f066453a700 1 -- 172.21.15.7:0/1149570641 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f065c142850 con 0x7f065c137710 2023-09-30T01:59:57.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.282+0000 7f065a7fc700 1 -- 172.21.15.7:0/1149570641 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f0654013690 con 0x7f065c137710 2023-09-30T01:59:57.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.282+0000 7f065a7fc700 1 --2- 172.21.15.7:0/1149570641 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f06480357e0 0x7f0648037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.282+0000 7f065a7fc700 1 -- 172.21.15.7:0/1149570641 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f06540480e0 con 0x7f065c137710 2023-09-30T01:59:57.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.282+0000 7f066453a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:57.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.282+0000 7f066453a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:57.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.282+0000 7f066453a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:57.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.282+0000 7f066453a700 1 -- 172.21.15.7:0/1149570641 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f065c1429e0 con 0x7f065c137710 2023-09-30T01:59:57.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.283+0000 7f06622d6700 1 --2- 172.21.15.7:0/1149570641 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f06480357e0 0x7f0648037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.283+0000 7f06622d6700 10 monclient: get_auth_request con 0x7f06480357e0 auth_method 0 2023-09-30T01:59:57.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.285+0000 7f06622d6700 1 --2- 172.21.15.7:0/1149570641 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f06480357e0 0x7f0648037c90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f064c00e280 tx=0x7f064c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:57.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.287+0000 7f065a7fc700 1 -- 172.21.15.7:0/1149570641 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f065401d460 con 0x7f065c137710 2023-09-30T01:59:57.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.287+0000 7f065a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:57.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.287+0000 7f065a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:57.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.483+0000 7f066453a700 1 -- 172.21.15.7:0/1149570641 --> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f065c11c230 con 0x7f06480357e0 2023-09-30T01:59:57.483 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.483+0000 7f065a7fc700 1 -- 172.21.15.7:0/1149570641 <== mgr.4105 v2:172.21.15.50:6824/86200 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f065c11c230 con 0x7f06480357e0 2023-09-30T01:59:57.485 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.485+0000 7f066453a700 1 -- 172.21.15.7:0/1149570641 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f06480357e0 msgr2=0x7f0648037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.485+0000 7f066453a700 1 --2- 172.21.15.7:0/1149570641 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f06480357e0 0x7f0648037c90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f064c00e280 tx=0x7f064c009040 comp rx=0 tx=0).stop 2023-09-30T01:59:57.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.485+0000 7f066453a700 10 monclient: shutdown 2023-09-30T01:59:57.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.485+0000 7f066453a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:57.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.485+0000 7f066453a700 1 -- 172.21.15.7:0/1149570641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f065c137710 msgr2=0x7f065c09e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.485+0000 7f066453a700 1 --2- 172.21.15.7:0/1149570641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f065c137710 0x7f065c09e6f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f065400d890 tx=0x7f0654008040 comp rx=0 tx=0).stop 2023-09-30T01:59:57.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.485+0000 7f066453a700 1 -- 172.21.15.7:0/1149570641 shutdown_connections 2023-09-30T01:59:57.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.485+0000 7f066453a700 1 --2- 172.21.15.7:0/1149570641 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f06480357e0 0x7f0648037c90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.485+0000 7f066453a700 1 --2- 172.21.15.7:0/1149570641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f065c137710 0x7f065c09e6f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.485+0000 7f066453a700 1 --2- 172.21.15.7:0/1149570641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f065c096a50 0x7f065c141dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.486 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.485+0000 7f066453a700 1 --2- 172.21.15.7:0/1149570641 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f065c096e20 0x7f065c142310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.485+0000 7f066453a700 1 -- 172.21.15.7:0/1149570641 >> 172.21.15.7:0/1149570641 conn(0x7f065c070cf0 msgr2=0x7f065c136140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:57.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.485+0000 7f066453a700 1 -- 172.21.15.7:0/1149570641 shutdown_connections 2023-09-30T01:59:57.487 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.485+0000 7f066453a700 1 -- 172.21.15.7:0/1149570641 wait complete. 2023-09-30T01:59:57.487 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 1 to deep-scrub 2023-09-30T01:59:57.497 INFO:tasks.ceph:Scrubbing osd.2 2023-09-30T01:59:57.497 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:59:57.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.618+0000 7fa34a019700 10 monclient: get_monmap_and_config 2023-09-30T01:59:57.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.618+0000 7fa34a019700 10 monclient: build_initial_monmap 2023-09-30T01:59:57.618 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.618+0000 7fa34a019700 10 monclient: monmap: 2023-09-30T01:59:57.618 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:57.618 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:57.619 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:57.619160+0000 2023-09-30T01:59:57.619 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:57.619160+0000 2023-09-30T01:59:57.619 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:57.619 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.619 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:57.619 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:57.619 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:57.619 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:57.619 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:57.620 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.618+0000 7fa34a019700 1 Processor -- start 2023-09-30T01:59:57.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.618+0000 7fa34a019700 1 -- start start 2023-09-30T01:59:57.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.618+0000 7fa34a019700 10 monclient: init 2023-09-30T01:59:57.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa34a019700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:57.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa34a019700 10 monclient: _add_conns ranks=[2,1,3,0,4] 2023-09-30T01:59:57.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa34a019700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa3440a2900 0x7fa344096a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.626 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa34a019700 10 monclient(hunting): picked mon.noname-b con 0x7fa3440a2900 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:57.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa34a019700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa344096f90 addr v1:172.21.15.50:6789/0 2023-09-30T01:59:57.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa34a019700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa344097380 0x7fa3441328c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa34a019700 10 monclient(hunting): picked mon.noname-a con 0x7fa344097380 addr v2:172.21.15.7:3300/0 2023-09-30T01:59:57.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa34a019700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3440991c0 con 0x7fa344096f90 2023-09-30T01:59:57.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa34a019700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:57.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa34a019700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa344098ea0 con 0x7fa344097380 2023-09-30T01:59:57.627 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa34a019700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:57.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa34a019700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3440a16d0 con 0x7fa3440a2900 2023-09-30T01:59:57.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa34a019700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:57.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa34a019700 10 monclient(hunting): authenticate will time out at 1486.684352s 2023-09-30T01:59:57.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa343fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa344097380 0x7fa3441328c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa3437fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa3440a2900 0x7fa344096a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa343fff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fa344097380 0x7fa3441328c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49380/0 (socket says 172.21.15.7:49380) 2023-09-30T01:59:57.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa3437fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa3440a2900 0x7fa344096a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41404/0 (socket says 172.21.15.7:41404) 2023-09-30T01:59:57.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa343fff700 1 -- 172.21.15.7:0/3916256872 learned_addr learned my addr 172.21.15.7:0/3916256872 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:57.628 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa343fff700 10 monclient(hunting): get_auth_request con 0x7fa344097380 auth_method 0 2023-09-30T01:59:57.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa343fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:57.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa343fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:57.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa343fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:57.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa3437fe700 10 monclient(hunting): get_auth_request con 0x7fa3440a2900 auth_method 0 2023-09-30T01:59:57.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa3437fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:57.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa3437fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:57.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa3437fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:57.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa343fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:57.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa343fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:57.629 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa343fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:57.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa3437fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:57.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa3437fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:57.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa3437fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:57.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa3437fe700 10 monclient(hunting): handle_auth_done global_id 4799 payload 274 2023-09-30T01:59:57.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa3437fe700 1 -- 172.21.15.7:0/3916256872 >> v2:172.21.15.7:3300/0 conn(0x7fa344097380 msgr2=0x7fa3441328c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:57.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa3437fe700 1 --2- 172.21.15.7:0/3916256872 >> v2:172.21.15.7:3300/0 conn(0x7fa344097380 0x7fa3441328c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa3437fe700 1 -- 172.21.15.7:0/3916256872 >> v1:172.21.15.50:6789/0 conn(0x7fa344096f90 legacy=0x7fa3441321b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:57.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa3437fe700 10 monclient: _finish_hunting 0 2023-09-30T01:59:57.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa3437fe700 1 monclient: found mon.noname-b 2023-09-30T01:59:57.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa3437fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:57.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa3437fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:57.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa3437fe700 1 -- 172.21.15.7:0/3916256872 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3440a2cd0 con 0x7fa3440a2900 2023-09-30T01:59:57.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.619+0000 7fa343fff700 1 --2- 172.21.15.7:0/3916256872 >> v2:172.21.15.7:3300/0 conn(0x7fa344097380 0x7fa3441328c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:57.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa3437fe700 1 -- 172.21.15.7:0/3916256872 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa3440a2900 msgr2=0x7fa344096a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:57.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa3437fe700 1 --2- 172.21.15.7:0/3916256872 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa3440a2900 0x7fa344096a50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa32c0098e0 tx=0x7fa32c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b280db2b897a0c9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:57.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa3427fc700 1 -- 172.21.15.7:0/3916256872 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c011840 con 0x7fa3440a2900 2023-09-30T01:59:57.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa3427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:57.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa3427fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:57.632 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa3427fc700 10 monclient: dump: 2023-09-30T01:59:57.632 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:57.632 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:57.632 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.632 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.632 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:57.633 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.633 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:57.633 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:57.633 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:57.633 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa3427fc700 10 monclient: _finish_auth 0 2023-09-30T01:59:57.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa3427fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:57.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa3427fc700 1 -- 172.21.15.7:0/3916256872 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32c0119a0 con 0x7fa3440a2900 2023-09-30T01:59:57.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa34a019700 5 monclient: authenticate success, global_id 4799 2023-09-30T01:59:57.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa34a019700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:57.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa3427fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:57.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa34a019700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:57.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa34a019700 4 set_mon_vals no callback set 2023-09-30T01:59:57.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa3427fc700 1 -- 172.21.15.7:0/3916256872 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c011d40 con 0x7fa3440a2900 2023-09-30T01:59:57.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa3427fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:57.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa3427fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:57.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa3427fc700 10 monclient: dump: 2023-09-30T01:59:57.634 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:57.635 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:57.635 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.635 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.635 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:57.635 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.635 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:57.635 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:57.635 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:57.635 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa34a019700 10 monclient: shutdown 2023-09-30T01:59:57.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa34a019700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:57.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa34a019700 1 -- 172.21.15.7:0/3916256872 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa3440a2900 msgr2=0x7fa344096a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa34a019700 1 --2- 172.21.15.7:0/3916256872 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa3440a2900 0x7fa344096a50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa32c0098e0 tx=0x7fa32c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa34a019700 1 -- 172.21.15.7:0/3916256872 shutdown_connections 2023-09-30T01:59:57.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa34a019700 1 --2- 172.21.15.7:0/3916256872 >> v2:172.21.15.7:3300/0 conn(0x7fa344097380 0x7fa3441328c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa34a019700 1 --2- 172.21.15.7:0/3916256872 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa3440a2900 0x7fa344096a50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa34a019700 1 -- 172.21.15.7:0/3916256872 >> 172.21.15.7:0/3916256872 conn(0x7fa3440a5cd0 msgr2=0x7fa3440a21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:57.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa34a019700 1 -- 172.21.15.7:0/3916256872 shutdown_connections 2023-09-30T01:59:57.636 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.622+0000 7fa34a019700 1 -- 172.21.15.7:0/3916256872 wait complete. 2023-09-30T01:59:57.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.623+0000 7fa34a019700 10 monclient: build_initial_monmap 2023-09-30T01:59:57.637 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.623+0000 7fa34a019700 10 monclient: monmap: 2023-09-30T01:59:57.637 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:57.637 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:57.637 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:57.637 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:57.637 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:57.637 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.637 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:57.637 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:57.638 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:57.638 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.623+0000 7fa34a019700 1 Processor -- start 2023-09-30T01:59:57.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.623+0000 7fa34a019700 1 -- start start 2023-09-30T01:59:57.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.623+0000 7fa34a019700 10 monclient: init 2023-09-30T01:59:57.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa34a019700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:57.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa34a019700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T01:59:57.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa34a019700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa344096f90 0x7fa34413e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa34a019700 10 monclient(hunting): picked mon.noname-b con 0x7fa344096f90 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:57.638 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa34a019700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa344097380 0x7fa34413da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa34a019700 10 monclient(hunting): picked mon.noname-a con 0x7fa344097380 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:57.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa34a019700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa3440a2900 0x7fa344141930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa34a019700 10 monclient(hunting): picked mon.noname-c con 0x7fa3440a2900 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:57.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa34a019700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:57.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa34a019700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa344123440 con 0x7fa344097380 2023-09-30T01:59:57.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa34a019700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:57.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa34a019700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa344122b10 con 0x7fa344096f90 2023-09-30T01:59:57.639 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa34a019700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa34a019700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3441207e0 con 0x7fa3440a2900 2023-09-30T01:59:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa34a019700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa34a019700 10 monclient(hunting): authenticate will time out at 1486.689226s 2023-09-30T01:59:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa343fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa3440a2900 0x7fa344141930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa342ffd700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa344097380 0x7fa34413da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa343fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa3440a2900 0x7fa344141930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41406/0 (socket says 172.21.15.7:41406) 2023-09-30T01:59:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa343fff700 1 -- 172.21.15.7:0/546350886 learned_addr learned my addr 172.21.15.7:0/546350886 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa343fff700 10 monclient(hunting): get_auth_request con 0x7fa3440a2900 auth_method 0 2023-09-30T01:59:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa343fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:57.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa343fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:57.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa343fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:57.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa342ffd700 10 monclient(hunting): get_auth_request con 0x7fa344097380 auth_method 0 2023-09-30T01:59:57.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa342ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:57.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa342ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:57.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa342ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:57.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa343fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:57.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa343fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:57.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa343fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:57.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa342ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:57.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa342ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:57.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa342ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:57.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa3437fe700 1 --2- 172.21.15.7:0/546350886 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa344096f90 0x7fa34413e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa342ffd700 10 monclient(hunting): handle_auth_done global_id 4773 payload 824 2023-09-30T01:59:57.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa342ffd700 1 -- 172.21.15.7:0/546350886 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa3440a2900 msgr2=0x7fa344141930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa342ffd700 1 --2- 172.21.15.7:0/546350886 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa3440a2900 0x7fa344141930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa342ffd700 1 -- 172.21.15.7:0/546350886 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa344096f90 msgr2=0x7fa34413e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa342ffd700 1 --2- 172.21.15.7:0/546350886 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa344096f90 0x7fa34413e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa342ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:59:57.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa342ffd700 1 monclient: found mon.noname-a 2023-09-30T01:59:57.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa342ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:57.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa342ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:57.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa342ffd700 1 -- 172.21.15.7:0/546350886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33c009ae0 con 0x7fa344097380 2023-09-30T01:59:57.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.624+0000 7fa343fff700 1 --2- 172.21.15.7:0/546350886 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fa3440a2900 0x7fa344141930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:57.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.626+0000 7fa342ffd700 1 -- 172.21.15.7:0/546350886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa344097380 msgr2=0x7fa34413da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:57.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.626+0000 7fa342ffd700 1 --2- 172.21.15.7:0/546350886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa344097380 0x7fa34413da70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa33400c4e0 tx=0x7fa334007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:57.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa33b7fe700 1 -- 172.21.15.7:0/546350886 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3340117a0 con 0x7fa344097380 2023-09-30T01:59:57.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa33b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:57.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa33b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:57.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa33b7fe700 10 monclient: dump: 2023-09-30T01:59:57.646 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:57.646 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:57.646 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.646 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.646 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:57.646 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.647 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:57.647 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:57.647 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:57.647 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa33b7fe700 10 monclient: _finish_auth 0 2023-09-30T01:59:57.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa33b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:57.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa33b7fe700 1 -- 172.21.15.7:0/546350886 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa334011920 con 0x7fa344097380 2023-09-30T01:59:57.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa34a019700 5 monclient: authenticate success, global_id 4773 2023-09-30T01:59:57.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa34a019700 10 monclient: _renew_subs 2023-09-30T01:59:57.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa34a019700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:57.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa34a019700 1 -- 172.21.15.7:0/546350886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa32c009420 con 0x7fa344097380 2023-09-30T01:59:57.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa33b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:57.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa33b7fe700 1 -- 172.21.15.7:0/546350886 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa334011ce0 con 0x7fa344097380 2023-09-30T01:59:57.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa33b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:57.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa33b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:57.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa33b7fe700 10 monclient: dump: 2023-09-30T01:59:57.648 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:57.648 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:57.648 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.649 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.649 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:57.649 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.649 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:57.649 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:57.649 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:57.649 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa341ffb700 4 set_mon_vals no callback set 2023-09-30T01:59:57.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa34a019700 10 monclient: _renew_subs 2023-09-30T01:59:57.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa34a019700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:57.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa34a019700 1 -- 172.21.15.7:0/546350886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa34413dfb0 con 0x7fa344097380 2023-09-30T01:59:57.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa34a019700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:57.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.627+0000 7fa34a019700 1 -- 172.21.15.7:0/546350886 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa34413e1d0 con 0x7fa344097380 2023-09-30T01:59:57.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.628+0000 7fa33b7fe700 1 -- 172.21.15.7:0/546350886 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fa334012630 con 0x7fa344097380 2023-09-30T01:59:57.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.628+0000 7fa342ffd700 1 --2- 172.21.15.7:0/546350886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa344097380 0x7fa34413da70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa33400c4e0 tx=0x7fa334007ea0 comp rx=0 tx=0).queue_received will delay after 1.69604e+09 on 0x7fa33404c2a0 mon_get_version_reply(handle=1 version=15) v2 2023-09-30T01:59:57.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.628+0000 7fa33b7fe700 1 --2- 172.21.15.7:0/546350886 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa324035860 0x7fa324037d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.628+0000 7fa33b7fe700 1 -- 172.21.15.7:0/546350886 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fa334047ab0 con 0x7fa344097380 2023-09-30T01:59:57.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.628+0000 7fa33b7fe700 1 --2- 172.21.15.7:0/546350886 >> [v2:172.21.15.7:6808/86602,v1:172.21.15.7:6809/86602] conn(0x7fa32403b6e0 0x7fa32403daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.628+0000 7fa33b7fe700 1 -- 172.21.15.7:0/546350886 --> [v2:172.21.15.7:6808/86602,v1:172.21.15.7:6809/86602] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa32403e1a0 con 0x7fa32403b6e0 2023-09-30T01:59:57.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.628+0000 7fa3437fe700 1 --2- 172.21.15.7:0/546350886 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa324035860 0x7fa324037d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.628+0000 7fa3437fe700 10 monclient: get_auth_request con 0x7fa324035860 auth_method 0 2023-09-30T01:59:57.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.628+0000 7fa343fff700 1 --2- 172.21.15.7:0/546350886 >> [v2:172.21.15.7:6808/86602,v1:172.21.15.7:6809/86602] conn(0x7fa32403b6e0 0x7fa32403daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.628+0000 7fa343fff700 10 monclient: get_auth_request con 0x7fa32403b6e0 auth_method 0 2023-09-30T01:59:57.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.631+0000 7fa3437fe700 1 --2- 172.21.15.7:0/546350886 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa324035860 0x7fa324037d10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00a010 tx=0x7fa32c016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:57.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.633+0000 7fa343fff700 1 --2- 172.21.15.7:0/546350886 >> [v2:172.21.15.7:6808/86602,v1:172.21.15.7:6809/86602] conn(0x7fa32403b6e0 0x7fa32403daf0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:57.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.633+0000 7fa33b7fe700 1 -- 172.21.15.7:0/546350886 <== osd.2 v2:172.21.15.7:6808/86602 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fa32403e1a0 con 0x7fa32403b6e0 2023-09-30T01:59:57.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.647+0000 7fa34a019700 1 -- 172.21.15.7:0/546350886 --> [v2:172.21.15.7:6808/86602,v1:172.21.15.7:6809/86602] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa344121e80 con 0x7fa32403b6e0 2023-09-30T01:59:57.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.647+0000 7fa33b7fe700 1 -- 172.21.15.7:0/546350886 <== osd.2 v2:172.21.15.7:6808/86602 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fa344121e80 con 0x7fa32403b6e0 2023-09-30T01:59:57.653 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-09-30T01:59:57.653 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:59:57.654 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-09-30T01:59:57.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.648+0000 7fa34a019700 1 -- 172.21.15.7:0/546350886 >> [v2:172.21.15.7:6808/86602,v1:172.21.15.7:6809/86602] conn(0x7fa32403b6e0 msgr2=0x7fa32403daf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.648+0000 7fa34a019700 1 --2- 172.21.15.7:0/546350886 >> [v2:172.21.15.7:6808/86602,v1:172.21.15.7:6809/86602] conn(0x7fa32403b6e0 0x7fa32403daf0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.648+0000 7fa34a019700 1 -- 172.21.15.7:0/546350886 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa324035860 msgr2=0x7fa324037d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.648+0000 7fa34a019700 1 --2- 172.21.15.7:0/546350886 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fa324035860 0x7fa324037d10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00a010 tx=0x7fa32c016000 comp rx=0 tx=0).stop 2023-09-30T01:59:57.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.648+0000 7fa34a019700 10 monclient: shutdown 2023-09-30T01:59:57.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.648+0000 7fa34a019700 20 monclient: shutdown canceling and discarding version request 1 2023-09-30T01:59:57.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.648+0000 7fa34a019700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:57.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.648+0000 7fa34a019700 1 -- 172.21.15.7:0/546350886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa344097380 msgr2=0x7fa34413da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.648+0000 7fa34a019700 1 --2- 172.21.15.7:0/546350886 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa344097380 0x7fa34413da70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa33400c4e0 tx=0x7fa334007ea0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.648+0000 7fa343fff700 1 -- 172.21.15.7:0/546350886 reap_dead start 2023-09-30T01:59:57.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.648+0000 7fa33b7fe700 1 -- 172.21.15.7:0/546350886 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (unknown 0 0 0) 0x7fa33404c2a0 con 0x7fa344097380 2023-09-30T01:59:57.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.648+0000 7fa33b7fe700 10 monclient: discarding stray monitor message mon_get_version_reply(handle=1 version=15) v2 2023-09-30T01:59:57.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.648+0000 7fa34a019700 1 -- 172.21.15.7:0/546350886 shutdown_connections 2023-09-30T01:59:57.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.648+0000 7fa34a019700 1 -- 172.21.15.7:0/546350886 >> 172.21.15.7:0/546350886 conn(0x7fa3440a5cd0 msgr2=0x7fa34409a900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:57.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.648+0000 7fa34a019700 1 -- 172.21.15.7:0/546350886 shutdown_connections 2023-09-30T01:59:57.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.648+0000 7fa34a019700 1 -- 172.21.15.7:0/546350886 wait complete. 2023-09-30T01:59:57.659 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-09-30T01:59:57.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.777+0000 7f6d928db700 10 monclient: get_monmap_and_config 2023-09-30T01:59:57.777 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.777+0000 7f6d928db700 10 monclient: build_initial_monmap 2023-09-30T01:59:57.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.777+0000 7f6d928db700 10 monclient: monmap: 2023-09-30T01:59:57.778 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:57.778 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:57.778 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:57.778723+0000 2023-09-30T01:59:57.778 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:57.778723+0000 2023-09-30T01:59:57.778 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:57.778 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.778 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:57.778 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:57.778 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:57.778 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:57.778 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:57.779 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.778+0000 7f6d928db700 1 Processor -- start 2023-09-30T01:59:57.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.778+0000 7f6d928db700 1 -- start start 2023-09-30T01:59:57.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.778+0000 7f6d928db700 10 monclient: init 2023-09-30T01:59:57.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.778+0000 7f6d928db700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:57.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.778+0000 7f6d928db700 10 monclient: _add_conns ranks=[4,2,1,3,0] 2023-09-30T01:59:57.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.778+0000 7f6d928db700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f6d8c098540 0x7f6d8c1329e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.778+0000 7f6d928db700 10 monclient(hunting): picked mon.noname-c con 0x7f6d8c098540 addr v2:172.21.15.50:3300/0 2023-09-30T01:59:57.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.778+0000 7f6d928db700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d8c098910 0x7f6d8c098ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d928db700 10 monclient(hunting): picked mon.noname-b con 0x7f6d8c098910 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:57.783 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d928db700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6d8c12ebc0 addr v1:172.21.15.50:6789/0 2023-09-30T01:59:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d928db700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8c069f10 con 0x7f6d8c12ebc0 2023-09-30T01:59:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d928db700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d928db700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d8c0a0f40 con 0x7f6d8c098540 2023-09-30T01:59:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d928db700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d928db700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c132f20 con 0x7f6d8c098910 2023-09-30T01:59:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d928db700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d928db700 10 monclient(hunting): authenticate will time out at 1486.844083s 2023-09-30T01:59:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d910d8700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d8c098910 0x7f6d8c098ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d910d8700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d8c098910 0x7f6d8c098ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41412/0 (socket says 172.21.15.7:41412) 2023-09-30T01:59:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d910d8700 1 -- 172.21.15.7:0/1607468375 learned_addr learned my addr 172.21.15.7:0/1607468375 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d910d8700 10 monclient(hunting): get_auth_request con 0x7f6d8c098910 auth_method 0 2023-09-30T01:59:57.784 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d910d8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:57.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d910d8700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:57.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d910d8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:57.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d918d9700 1 --2- 172.21.15.7:0/1607468375 >> v2:172.21.15.50:3300/0 conn(0x7f6d8c098540 0x7f6d8c1329e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d918d9700 10 monclient(hunting): get_auth_request con 0x7f6d8c098540 auth_method 0 2023-09-30T01:59:57.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d918d9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:57.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d918d9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:57.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d918d9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:57.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d910d8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:57.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d910d8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:57.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d910d8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:57.785 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d918d9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d918d9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d918d9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d910d8700 10 monclient(hunting): handle_auth_done global_id 4805 payload 274 2023-09-30T01:59:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d910d8700 1 -- 172.21.15.7:0/1607468375 >> v2:172.21.15.50:3300/0 conn(0x7f6d8c098540 msgr2=0x7f6d8c1329e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d910d8700 1 --2- 172.21.15.7:0/1607468375 >> v2:172.21.15.50:3300/0 conn(0x7f6d8c098540 0x7f6d8c1329e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d910d8700 1 -- 172.21.15.7:0/1607468375 >> v1:172.21.15.50:6789/0 conn(0x7f6d8c12ebc0 legacy=0x7f6d8c12f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d910d8700 10 monclient: _finish_hunting 0 2023-09-30T01:59:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d910d8700 1 monclient: found mon.noname-b 2023-09-30T01:59:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d910d8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d910d8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.779+0000 7f6d910d8700 1 -- 172.21.15.7:0/1607468375 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8c096a50 con 0x7f6d8c098910 2023-09-30T01:59:57.786 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.781+0000 7f6d910d8700 1 -- 172.21.15.7:0/1607468375 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d8c098910 msgr2=0x7f6d8c098ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:57.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.781+0000 7f6d910d8700 1 --2- 172.21.15.7:0/1607468375 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d8c098910 0x7f6d8c098ce0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6d800099b0 tx=0x7f6d80009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a80ab7120ed108f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:57.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.781+0000 7f6d908d7700 1 -- 172.21.15.7:0/1607468375 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d80009ea0 con 0x7f6d8c098910 2023-09-30T01:59:57.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.781+0000 7f6d908d7700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:57.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.781+0000 7f6d908d7700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:57.787 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.781+0000 7f6d908d7700 10 monclient: dump: 2023-09-30T01:59:57.787 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:57.787 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:57.787 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.787 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.787 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:57.787 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:57.788 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.781+0000 7f6d908d7700 10 monclient: _finish_auth 0 2023-09-30T01:59:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.781+0000 7f6d908d7700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.781+0000 7f6d928db700 5 monclient: authenticate success, global_id 4805 2023-09-30T01:59:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.781+0000 7f6d928db700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.781+0000 7f6d908d7700 1 -- 172.21.15.7:0/1607468375 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d8000ba30 con 0x7f6d8c098910 2023-09-30T01:59:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.781+0000 7f6d908d7700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.781+0000 7f6d928db700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:57.788 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.781+0000 7f6d928db700 4 set_mon_vals no callback set 2023-09-30T01:59:57.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.781+0000 7f6d908d7700 1 -- 172.21.15.7:0/1607468375 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8000bdf0 con 0x7f6d8c098910 2023-09-30T01:59:57.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.781+0000 7f6d908d7700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:57.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.781+0000 7f6d908d7700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:57.789 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.781+0000 7f6d908d7700 10 monclient: dump: 2023-09-30T01:59:57.789 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:57.789 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:57.789 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.789 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.789 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:57.789 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.789 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:57.789 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:57.790 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:57.790 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.782+0000 7f6d928db700 10 monclient: shutdown 2023-09-30T01:59:57.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.782+0000 7f6d928db700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:57.790 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.782+0000 7f6d928db700 1 -- 172.21.15.7:0/1607468375 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d8c098910 msgr2=0x7f6d8c098ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.782+0000 7f6d928db700 1 --2- 172.21.15.7:0/1607468375 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d8c098910 0x7f6d8c098ce0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6d800099b0 tx=0x7f6d80009250 comp rx=0 tx=0).stop 2023-09-30T01:59:57.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.782+0000 7f6d928db700 1 -- 172.21.15.7:0/1607468375 shutdown_connections 2023-09-30T01:59:57.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.782+0000 7f6d928db700 1 --2- 172.21.15.7:0/1607468375 >> v2:172.21.15.50:3300/0 conn(0x7f6d8c098540 0x7f6d8c1329e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.782+0000 7f6d928db700 1 --2- 172.21.15.7:0/1607468375 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d8c098910 0x7f6d8c098ce0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.791 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.782+0000 7f6d928db700 1 -- 172.21.15.7:0/1607468375 >> 172.21.15.7:0/1607468375 conn(0x7f6d8c071830 msgr2=0x7f6d8c0a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:57.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.782+0000 7f6d928db700 1 -- 172.21.15.7:0/1607468375 shutdown_connections 2023-09-30T01:59:57.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.782+0000 7f6d928db700 1 -- 172.21.15.7:0/1607468375 wait complete. 2023-09-30T01:59:57.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d928db700 10 monclient: build_initial_monmap 2023-09-30T01:59:57.792 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d928db700 10 monclient: monmap: 2023-09-30T01:59:57.792 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:57.792 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:57.793 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:57.793 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:57.793 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:57.793 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.793 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:57.794 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:57.794 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:57.794 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d928db700 1 Processor -- start 2023-09-30T01:59:57.794 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d928db700 1 -- start start 2023-09-30T01:59:57.795 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d928db700 10 monclient: init 2023-09-30T01:59:57.799 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d928db700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:57.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d928db700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:59:57.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d928db700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d8c098540 0x7f6d8c1352e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d928db700 10 monclient(hunting): picked mon.noname-c con 0x7f6d8c098540 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:57.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d928db700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d8c098910 0x7f6d8c12dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.800 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d928db700 10 monclient(hunting): picked mon.noname-b con 0x7f6d8c098910 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:57.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d928db700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d8c12ebc0 0x7f6d8c099300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d928db700 10 monclient(hunting): picked mon.noname-a con 0x7f6d8c12ebc0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:57.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d928db700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:57.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d928db700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c09b880 con 0x7f6d8c12ebc0 2023-09-30T01:59:57.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d928db700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:57.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d928db700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c0a5070 con 0x7f6d8c098910 2023-09-30T01:59:57.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d928db700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:57.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d928db700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c09a990 con 0x7f6d8c098540 2023-09-30T01:59:57.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d928db700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:57.801 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d928db700 10 monclient(hunting): authenticate will time out at 1486.848847s 2023-09-30T01:59:57.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d918d9700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d8c098540 0x7f6d8c1352e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d918d9700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d8c098540 0x7f6d8c1352e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41422/0 (socket says 172.21.15.7:41422) 2023-09-30T01:59:57.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d918d9700 1 -- 172.21.15.7:0/4029902450 learned_addr learned my addr 172.21.15.7:0/4029902450 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:57.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d918d9700 10 monclient(hunting): get_auth_request con 0x7f6d8c098540 auth_method 0 2023-09-30T01:59:57.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d918d9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:57.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d918d9700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:57.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d918d9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:57.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.783+0000 7f6d920da700 1 --2- 172.21.15.7:0/4029902450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d8c12ebc0 0x7f6d8c099300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d920da700 10 monclient(hunting): get_auth_request con 0x7f6d8c12ebc0 auth_method 0 2023-09-30T01:59:57.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d920da700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:57.802 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d920da700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:57.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d920da700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:57.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d910d8700 1 --2- 172.21.15.7:0/4029902450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d8c098910 0x7f6d8c12dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d918d9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:57.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d918d9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:57.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d918d9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:57.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d910d8700 10 monclient(hunting): get_auth_request con 0x7f6d8c098910 auth_method 0 2023-09-30T01:59:57.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d910d8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:57.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d910d8700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:57.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d910d8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:57.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d920da700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:57.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d920da700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:57.803 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d920da700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:57.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d910d8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:57.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d910d8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:57.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d910d8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:57.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d918d9700 10 monclient(hunting): handle_auth_done global_id 4808 payload 824 2023-09-30T01:59:57.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d918d9700 1 -- 172.21.15.7:0/4029902450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d8c098910 msgr2=0x7f6d8c12dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d918d9700 1 --2- 172.21.15.7:0/4029902450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d8c098910 0x7f6d8c12dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d918d9700 1 -- 172.21.15.7:0/4029902450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d8c12ebc0 msgr2=0x7f6d8c099300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d918d9700 1 --2- 172.21.15.7:0/4029902450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d8c12ebc0 0x7f6d8c099300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d918d9700 10 monclient: _finish_hunting 0 2023-09-30T01:59:57.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d918d9700 1 monclient: found mon.noname-c 2023-09-30T01:59:57.804 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d918d9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:57.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d918d9700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:59:57.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.784+0000 7f6d918d9700 1 -- 172.21.15.7:0/4029902450 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d800094f0 con 0x7f6d8c098540 2023-09-30T01:59:57.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d918d9700 1 -- 172.21.15.7:0/4029902450 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d8c098540 msgr2=0x7f6d8c1352e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:57.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d918d9700 1 --2- 172.21.15.7:0/4029902450 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d8c098540 0x7f6d8c1352e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c00e930 tx=0x7f6d7c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:57.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d89ffb700 1 -- 172.21.15.7:0/4029902450 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c013890 con 0x7f6d8c098540 2023-09-30T01:59:57.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d89ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:57.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d89ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:59:57.805 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d89ffb700 10 monclient: dump: 2023-09-30T01:59:57.805 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:57.805 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:57.805 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.806 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.806 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:57.806 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.806 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:57.806 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:57.806 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:57.806 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d89ffb700 10 monclient: _finish_auth 0 2023-09-30T01:59:57.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d89ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:57.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d928db700 5 monclient: authenticate success, global_id 4808 2023-09-30T01:59:57.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d89ffb700 1 -- 172.21.15.7:0/4029902450 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d7c0139f0 con 0x7f6d8c098540 2023-09-30T01:59:57.806 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d928db700 10 monclient: _renew_subs 2023-09-30T01:59:57.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d928db700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:57.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d928db700 1 -- 172.21.15.7:0/4029902450 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8c0a05c0 con 0x7f6d8c098540 2023-09-30T01:59:57.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d89ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:57.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d89ffb700 1 -- 172.21.15.7:0/4029902450 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c013db0 con 0x7f6d8c098540 2023-09-30T01:59:57.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d89ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:57.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d89ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:57.807 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d89ffb700 10 monclient: dump: 2023-09-30T01:59:57.807 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:57.807 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:57.807 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.807 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:57.808 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:57.808 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:57.808 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:57.808 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:57.808 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:57.808 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:57.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d928db700 10 monclient: _renew_subs 2023-09-30T01:59:57.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d928db700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:57.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d8bfff700 4 set_mon_vals no callback set 2023-09-30T01:59:57.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.786+0000 7f6d928db700 1 -- 172.21.15.7:0/4029902450 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8c1425f0 con 0x7f6d8c098540 2023-09-30T01:59:57.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.787+0000 7f6d89ffb700 1 -- 172.21.15.7:0/4029902450 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f6d7c00c5f0 con 0x7f6d8c098540 2023-09-30T01:59:57.808 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.787+0000 7f6d928db700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:57.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.787+0000 7f6d928db700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:57.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.787+0000 7f6d928db700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:57.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.787+0000 7f6d928db700 1 -- 172.21.15.7:0/4029902450 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d8c09c080 con 0x7f6d8c098540 2023-09-30T01:59:57.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.787+0000 7f6d89ffb700 1 --2- 172.21.15.7:0/4029902450 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6d780357e0 0x7f6d78037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:57.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.787+0000 7f6d89ffb700 1 -- 172.21.15.7:0/4029902450 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f6d7c047de0 con 0x7f6d8c098540 2023-09-30T01:59:57.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.788+0000 7f6d910d8700 1 --2- 172.21.15.7:0/4029902450 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6d780357e0 0x7f6d78037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:57.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.788+0000 7f6d910d8700 10 monclient: get_auth_request con 0x7f6d780357e0 auth_method 0 2023-09-30T01:59:57.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.790+0000 7f6d910d8700 1 --2- 172.21.15.7:0/4029902450 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6d780357e0 0x7f6d78037c90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6d800099b0 tx=0x7f6d80016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:57.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.792+0000 7f6d89ffb700 1 -- 172.21.15.7:0/4029902450 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6d7c023700 con 0x7f6d8c098540 2023-09-30T01:59:57.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.792+0000 7f6d89ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:57.809 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.792+0000 7f6d89ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:57.986 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.986+0000 7f6d928db700 1 -- 172.21.15.7:0/4029902450 --> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d8c064c10 con 0x7f6d780357e0 2023-09-30T01:59:57.987 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.987+0000 7f6d89ffb700 1 -- 172.21.15.7:0/4029902450 <== mgr.4105 v2:172.21.15.50:6824/86200 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6d8c064c10 con 0x7f6d780357e0 2023-09-30T01:59:57.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.989+0000 7f6d928db700 1 -- 172.21.15.7:0/4029902450 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6d780357e0 msgr2=0x7f6d78037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.989+0000 7f6d928db700 1 --2- 172.21.15.7:0/4029902450 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6d780357e0 0x7f6d78037c90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6d800099b0 tx=0x7f6d80016000 comp rx=0 tx=0).stop 2023-09-30T01:59:57.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.989+0000 7f6d928db700 10 monclient: shutdown 2023-09-30T01:59:57.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.989+0000 7f6d928db700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:57.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.989+0000 7f6d928db700 1 -- 172.21.15.7:0/4029902450 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d8c098540 msgr2=0x7f6d8c1352e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:57.989 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.989+0000 7f6d928db700 1 --2- 172.21.15.7:0/4029902450 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d8c098540 0x7f6d8c1352e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c00e930 tx=0x7f6d7c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.989+0000 7f6d928db700 1 -- 172.21.15.7:0/4029902450 shutdown_connections 2023-09-30T01:59:57.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.989+0000 7f6d928db700 1 --2- 172.21.15.7:0/4029902450 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f6d780357e0 0x7f6d78037c90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.989+0000 7f6d928db700 1 --2- 172.21.15.7:0/4029902450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d8c12ebc0 0x7f6d8c099300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.989+0000 7f6d928db700 1 --2- 172.21.15.7:0/4029902450 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f6d8c098540 0x7f6d8c1352e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.989+0000 7f6d928db700 1 --2- 172.21.15.7:0/4029902450 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d8c098910 0x7f6d8c12dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:57.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.989+0000 7f6d928db700 1 -- 172.21.15.7:0/4029902450 >> 172.21.15.7:0/4029902450 conn(0x7f6d8c071830 msgr2=0x7f6d8c0a1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:57.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.989+0000 7f6d928db700 1 -- 172.21.15.7:0/4029902450 shutdown_connections 2023-09-30T01:59:57.990 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:57.989+0000 7f6d928db700 1 -- 172.21.15.7:0/4029902450 wait complete. 2023-09-30T01:59:57.990 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 2 to deep-scrub 2023-09-30T01:59:58.001 INFO:tasks.ceph:Scrubbing osd.3 2023-09-30T01:59:58.001 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:59:58.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.122+0000 7f4130385700 10 monclient: get_monmap_and_config 2023-09-30T01:59:58.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.122+0000 7f4130385700 10 monclient: build_initial_monmap 2023-09-30T01:59:58.122 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.122+0000 7f4130385700 10 monclient: monmap: 2023-09-30T01:59:58.122 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:58.122 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:58.122 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:58.123383+0000 2023-09-30T01:59:58.123 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:58.123383+0000 2023-09-30T01:59:58.123 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:58.123 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:58.123 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:58.123 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:58.123 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:58.123 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:58.123 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:58.123 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:58.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.122+0000 7f4130385700 1 Processor -- start 2023-09-30T01:59:58.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.122+0000 7f4130385700 1 -- start start 2023-09-30T01:59:58.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.122+0000 7f4130385700 10 monclient: init 2023-09-30T01:59:58.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f4130385700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:58.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f4130385700 10 monclient: _add_conns ranks=[3,4,0,2,1] 2023-09-30T01:59:58.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f4130385700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4128132280 0x7f4128132650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f4130385700 10 monclient(hunting): picked mon.noname-a con 0x7f4128132280 addr v2:172.21.15.7:3300/0 2023-09-30T01:59:58.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f4130385700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f41280a2270 0x7f41280a2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.126 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f4130385700 10 monclient(hunting): picked mon.noname-c con 0x7f41280a2270 addr v2:172.21.15.50:3300/0 2023-09-30T01:59:58.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f4130385700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f41280a2bb0 addr v1:172.21.15.7:6789/0 2023-09-30T01:59:58.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f4130385700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f412809a880 con 0x7f41280a2bb0 2023-09-30T01:59:58.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f4130385700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:58.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f4130385700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f412809a510 con 0x7f4128132280 2023-09-30T01:59:58.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f4130385700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:58.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f4130385700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41280996c0 con 0x7f41280a2270 2023-09-30T01:59:58.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f4130385700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:58.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f4130385700 10 monclient(hunting): authenticate will time out at 1487.188595s 2023-09-30T01:59:58.127 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f412e121700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4128132280 0x7f4128132650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f412e121700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f4128132280 0x7f4128132650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49398/0 (socket says 172.21.15.7:49398) 2023-09-30T01:59:58.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f412e121700 1 -- 172.21.15.7:0/867437477 learned_addr learned my addr 172.21.15.7:0/867437477 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:58.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f412e121700 10 monclient(hunting): get_auth_request con 0x7f4128132280 auth_method 0 2023-09-30T01:59:58.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f412e121700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:58.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f412e121700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:58.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f412e121700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:58.128 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.123+0000 7f412d920700 1 --2- 172.21.15.7:0/867437477 >> v2:172.21.15.50:3300/0 conn(0x7f41280a2270 0x7f41280a2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.124+0000 7f412d920700 10 monclient(hunting): get_auth_request con 0x7f41280a2270 auth_method 0 2023-09-30T01:59:58.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.124+0000 7f412d920700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:58.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.124+0000 7f412d920700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:58.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.124+0000 7f412d920700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:58.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.124+0000 7f412d920700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:58.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.124+0000 7f412d920700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:58.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.124+0000 7f412d920700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:58.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.124+0000 7f412d920700 10 monclient(hunting): handle_auth_done global_id 4645 payload 274 2023-09-30T01:59:58.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.124+0000 7f412d920700 1 -- 172.21.15.7:0/867437477 >> v2:172.21.15.7:3300/0 conn(0x7f4128132280 msgr2=0x7f4128132650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:58.129 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.124+0000 7f412d920700 1 --2- 172.21.15.7:0/867437477 >> v2:172.21.15.7:3300/0 conn(0x7f4128132280 0x7f4128132650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.124+0000 7f412d920700 1 -- 172.21.15.7:0/867437477 >> v1:172.21.15.7:6789/0 conn(0x7f41280a2bb0 legacy=0x7f4128097070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:58.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.124+0000 7f412d920700 10 monclient: _finish_hunting 0 2023-09-30T01:59:58.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.124+0000 7f412d920700 1 monclient: found mon.noname-c 2023-09-30T01:59:58.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.124+0000 7f412d920700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:58.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.124+0000 7f412d920700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.50:3300/0 2023-09-30T01:59:58.130 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.124+0000 7f412d920700 1 -- 172.21.15.7:0/867437477 --> v2:172.21.15.50:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412809a230 con 0x7f41280a2270 2023-09-30T01:59:58.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.126+0000 7f412d920700 1 -- 172.21.15.7:0/867437477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41280a2270 msgr2=0x7f41280a2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:58.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.126+0000 7f412d920700 1 --2- 172.21.15.7:0/867437477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41280a2270 0x7f41280a2670 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f412806ac30 tx=0x7f41180091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9cda91eb3012fdd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:58.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.126+0000 7f412d11f700 1 -- 172.21.15.7:0/867437477 <== mon.1 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411800b880 con 0x7f41280a2270 2023-09-30T01:59:58.137 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.126+0000 7f412d11f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:58.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.126+0000 7f412d11f700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:59:58.138 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.126+0000 7f412d11f700 10 monclient: dump: 2023-09-30T01:59:58.138 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:58.138 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:58.138 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.138 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.138 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:58.138 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:58.138 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:58.138 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:58.138 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:58.139 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:58.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.126+0000 7f412d11f700 10 monclient: _finish_auth 0 2023-09-30T01:59:58.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.126+0000 7f412d11f700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:58.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.126+0000 7f4130385700 5 monclient: authenticate success, global_id 4645 2023-09-30T01:59:58.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.126+0000 7f4130385700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:58.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.126+0000 7f412d11f700 1 -- 172.21.15.7:0/867437477 <== mon.1 v2:172.21.15.50:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411800b9e0 con 0x7f41280a2270 2023-09-30T01:59:58.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.126+0000 7f412d11f700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:58.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.126+0000 7f4130385700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:58.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.126+0000 7f4130385700 4 set_mon_vals no callback set 2023-09-30T01:59:58.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.126+0000 7f412d11f700 1 -- 172.21.15.7:0/867437477 <== mon.1 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411800bd80 con 0x7f41280a2270 2023-09-30T01:59:58.139 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.126+0000 7f412d11f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:58.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.126+0000 7f412d11f700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T01:59:58.140 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.126+0000 7f412d11f700 10 monclient: dump: 2023-09-30T01:59:58.140 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:58.140 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:58.140 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.140 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.140 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:58.140 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:58.140 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:58.140 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:58.140 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:58.141 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:58.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.127+0000 7f4130385700 10 monclient: shutdown 2023-09-30T01:59:58.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.127+0000 7f4130385700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:58.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.127+0000 7f4130385700 1 -- 172.21.15.7:0/867437477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41280a2270 msgr2=0x7f41280a2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.127+0000 7f4130385700 1 --2- 172.21.15.7:0/867437477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41280a2270 0x7f41280a2670 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f412806ac30 tx=0x7f41180091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.127+0000 7f4130385700 1 -- 172.21.15.7:0/867437477 shutdown_connections 2023-09-30T01:59:58.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.127+0000 7f4130385700 1 --2- 172.21.15.7:0/867437477 >> v2:172.21.15.7:3300/0 conn(0x7f4128132280 0x7f4128132650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.127+0000 7f4130385700 1 --2- 172.21.15.7:0/867437477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41280a2270 0x7f41280a2670 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.127+0000 7f4130385700 1 -- 172.21.15.7:0/867437477 >> 172.21.15.7:0/867437477 conn(0x7f41280677e0 msgr2=0x7f4128067be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:58.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.127+0000 7f4130385700 1 -- 172.21.15.7:0/867437477 shutdown_connections 2023-09-30T01:59:58.141 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.127+0000 7f4130385700 1 -- 172.21.15.7:0/867437477 wait complete. 2023-09-30T01:59:58.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.127+0000 7f4130385700 10 monclient: build_initial_monmap 2023-09-30T01:59:58.142 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.127+0000 7f4130385700 10 monclient: monmap: 2023-09-30T01:59:58.142 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:58.142 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:58.142 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:58.142 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:58.142 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:58.142 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:58.142 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:58.142 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:58.143 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:58.143 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:58.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f4130385700 1 Processor -- start 2023-09-30T01:59:58.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f4130385700 1 -- start start 2023-09-30T01:59:58.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f4130385700 10 monclient: init 2023-09-30T01:59:58.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f4130385700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:58.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f4130385700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:59:58.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f4130385700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41280a2270 0x7f412811d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f4130385700 10 monclient(hunting): picked mon.noname-c con 0x7f41280a2270 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:58.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f4130385700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4128132280 0x7f412811d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f4130385700 10 monclient(hunting): picked mon.noname-b con 0x7f4128132280 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:58.143 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f4130385700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f412811faf0 0x7f412811dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f4130385700 10 monclient(hunting): picked mon.noname-a con 0x7f412811faf0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:58.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f4130385700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:58.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f4130385700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41280a2040 con 0x7f412811faf0 2023-09-30T01:59:58.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f4130385700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:58.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f4130385700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41280a12b0 con 0x7f4128132280 2023-09-30T01:59:58.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f4130385700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:58.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f4130385700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4128120d50 con 0x7f41280a2270 2023-09-30T01:59:58.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f4130385700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:58.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f4130385700 10 monclient(hunting): authenticate will time out at 1487.193664s 2023-09-30T01:59:58.144 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f412e121700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41280a2270 0x7f412811d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f412e121700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41280a2270 0x7f412811d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41432/0 (socket says 172.21.15.7:41432) 2023-09-30T01:59:58.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f412e922700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f412811faf0 0x7f412811dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f412e121700 1 -- 172.21.15.7:0/1629939650 learned_addr learned my addr 172.21.15.7:0/1629939650 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:58.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f412e121700 10 monclient(hunting): get_auth_request con 0x7f41280a2270 auth_method 0 2023-09-30T01:59:58.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f412e121700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:58.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f412e121700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:58.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f412e121700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:58.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f412e922700 10 monclient(hunting): get_auth_request con 0x7f412811faf0 auth_method 0 2023-09-30T01:59:58.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f412e922700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:58.145 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f412e922700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:58.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f412e922700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:58.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f412e922700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:58.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f412e922700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:58.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f412e922700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:58.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f412e121700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:58.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f412e121700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:58.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f412e121700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:58.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.128+0000 7f412d920700 1 --2- 172.21.15.7:0/1629939650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4128132280 0x7f412811d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.129+0000 7f412d920700 10 monclient(hunting): get_auth_request con 0x7f4128132280 auth_method 0 2023-09-30T01:59:58.146 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.129+0000 7f412d920700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:58.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.129+0000 7f412d920700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:58.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.129+0000 7f412d920700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:58.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.129+0000 7f412e922700 10 monclient(hunting): handle_auth_done global_id 4782 payload 824 2023-09-30T01:59:58.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.129+0000 7f412e922700 1 -- 172.21.15.7:0/1629939650 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41280a2270 msgr2=0x7f412811d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.129+0000 7f412e922700 1 --2- 172.21.15.7:0/1629939650 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41280a2270 0x7f412811d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.129+0000 7f412e922700 1 -- 172.21.15.7:0/1629939650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4128132280 msgr2=0x7f412811d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.147 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.129+0000 7f412e922700 1 --2- 172.21.15.7:0/1629939650 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4128132280 0x7f412811d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.129+0000 7f412e922700 10 monclient: _finish_hunting 0 2023-09-30T01:59:58.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.129+0000 7f412e922700 1 monclient: found mon.noname-a 2023-09-30T01:59:58.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.129+0000 7f412e922700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:58.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.129+0000 7f412e922700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:58.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.129+0000 7f412e922700 1 -- 172.21.15.7:0/1629939650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4118009420 con 0x7f412811faf0 2023-09-30T01:59:58.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.129+0000 7f412e121700 1 --2- 172.21.15.7:0/1629939650 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f41280a2270 0x7f412811d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:58.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f412e922700 1 -- 172.21.15.7:0/1629939650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f412811faf0 msgr2=0x7f412811dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:58.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f412e922700 1 --2- 172.21.15.7:0/1629939650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f412811faf0 0x7f412811dd50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f412000c810 tx=0x7f4120002620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:58.148 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f411e7fc700 1 -- 172.21.15.7:0/1629939650 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4120012840 con 0x7f412811faf0 2023-09-30T01:59:58.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f411e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:58.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f411e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:58.149 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f411e7fc700 10 monclient: dump: 2023-09-30T01:59:58.149 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:58.149 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:58.149 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.149 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.149 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:58.150 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f411e7fc700 10 monclient: _finish_auth 0 2023-09-30T01:59:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f411e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f411e7fc700 1 -- 172.21.15.7:0/1629939650 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41200129a0 con 0x7f412811faf0 2023-09-30T01:59:58.150 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f4130385700 5 monclient: authenticate success, global_id 4782 2023-09-30T01:59:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f4130385700 10 monclient: _renew_subs 2023-09-30T01:59:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f4130385700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f4130385700 1 -- 172.21.15.7:0/1629939650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4128130ad0 con 0x7f412811faf0 2023-09-30T01:59:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f411e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f411e7fc700 1 -- 172.21.15.7:0/1629939650 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4120012d60 con 0x7f412811faf0 2023-09-30T01:59:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f411e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f411e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f411e7fc700 10 monclient: dump: 2023-09-30T01:59:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:58.151 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:58.152 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f412c91e700 4 set_mon_vals no callback set 2023-09-30T01:59:58.152 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f4130385700 10 monclient: _renew_subs 2023-09-30T01:59:58.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f4130385700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:58.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.131+0000 7f4130385700 1 -- 172.21.15.7:0/1629939650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f412811e290 con 0x7f412811faf0 2023-09-30T01:59:58.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.132+0000 7f4130385700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:58.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.132+0000 7f4130385700 1 -- 172.21.15.7:0/1629939650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f41280a2760 con 0x7f412811faf0 2023-09-30T01:59:58.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.132+0000 7f411e7fc700 1 -- 172.21.15.7:0/1629939650 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f4120011590 con 0x7f412811faf0 2023-09-30T01:59:58.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.132+0000 7f411e7fc700 1 --2- 172.21.15.7:0/1629939650 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f410c0357e0 0x7f410c037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.132+0000 7f411e7fc700 1 -- 172.21.15.7:0/1629939650 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f4120047800 con 0x7f412811faf0 2023-09-30T01:59:58.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.132+0000 7f411e7fc700 1 --2- 172.21.15.7:0/1629939650 >> [v2:172.21.15.50:6804/86633,v1:172.21.15.50:6807/86633] conn(0x7f410c03b660 0x7f410c03da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.132+0000 7f411e7fc700 1 -- 172.21.15.7:0/1629939650 --> [v2:172.21.15.50:6804/86633,v1:172.21.15.50:6807/86633] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f410c03e120 con 0x7f410c03b660 2023-09-30T01:59:58.153 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.132+0000 7f411e7fc700 1 -- 172.21.15.7:0/1629939650 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4120049d90 con 0x7f412811faf0 2023-09-30T01:59:58.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.132+0000 7f411e7fc700 10 monclient: handle_get_version_reply finishing 1 version 15 2023-09-30T01:59:58.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.133+0000 7f412e121700 1 --2- 172.21.15.7:0/1629939650 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f410c0357e0 0x7f410c037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.133+0000 7f412e121700 10 monclient: get_auth_request con 0x7f410c0357e0 auth_method 0 2023-09-30T01:59:58.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.133+0000 7f412d920700 1 --2- 172.21.15.7:0/1629939650 >> [v2:172.21.15.50:6804/86633,v1:172.21.15.50:6807/86633] conn(0x7f410c03b660 0x7f410c03da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.133+0000 7f412d920700 10 monclient: get_auth_request con 0x7f410c03b660 auth_method 0 2023-09-30T01:59:58.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.135+0000 7f412e121700 1 --2- 172.21.15.7:0/1629939650 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f410c0357e0 0x7f410c037c90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4124006b00 tx=0x7f4124008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:58.154 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.137+0000 7f412d920700 1 --2- 172.21.15.7:0/1629939650 >> [v2:172.21.15.50:6804/86633,v1:172.21.15.50:6807/86633] conn(0x7f410c03b660 0x7f410c03da70 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:58.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.138+0000 7f411e7fc700 1 -- 172.21.15.7:0/1629939650 <== osd.3 v2:172.21.15.50:6804/86633 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f410c03e120 con 0x7f410c03b660 2023-09-30T01:59:58.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.152+0000 7f4130385700 1 -- 172.21.15.7:0/1629939650 --> [v2:172.21.15.50:6804/86633,v1:172.21.15.50:6807/86633] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4128130a70 con 0x7f410c03b660 2023-09-30T01:59:58.155 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.153+0000 7f411e7fc700 1 -- 172.21.15.7:0/1629939650 <== osd.3 v2:172.21.15.50:6804/86633 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f4128130a70 con 0x7f410c03b660 2023-09-30T01:59:58.155 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-09-30T01:59:58.156 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:59:58.156 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-09-30T01:59:58.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.153+0000 7f4130385700 1 -- 172.21.15.7:0/1629939650 >> [v2:172.21.15.50:6804/86633,v1:172.21.15.50:6807/86633] conn(0x7f410c03b660 msgr2=0x7f410c03da70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.153+0000 7f4130385700 1 --2- 172.21.15.7:0/1629939650 >> [v2:172.21.15.50:6804/86633,v1:172.21.15.50:6807/86633] conn(0x7f410c03b660 0x7f410c03da70 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.153+0000 7f4130385700 1 -- 172.21.15.7:0/1629939650 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f410c0357e0 msgr2=0x7f410c037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.153+0000 7f4130385700 1 --2- 172.21.15.7:0/1629939650 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f410c0357e0 0x7f410c037c90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4124006b00 tx=0x7f4124008040 comp rx=0 tx=0).stop 2023-09-30T01:59:58.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.153+0000 7f4130385700 10 monclient: shutdown 2023-09-30T01:59:58.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.153+0000 7f4130385700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:58.157 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.153+0000 7f4130385700 1 -- 172.21.15.7:0/1629939650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f412811faf0 msgr2=0x7f412811dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.153+0000 7f4130385700 1 --2- 172.21.15.7:0/1629939650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f412811faf0 0x7f412811dd50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f412000c810 tx=0x7f4120002620 comp rx=0 tx=0).stop 2023-09-30T01:59:58.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.153+0000 7f412e922700 1 -- 172.21.15.7:0/1629939650 reap_dead start 2023-09-30T01:59:58.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.153+0000 7f4130385700 1 -- 172.21.15.7:0/1629939650 shutdown_connections 2023-09-30T01:59:58.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.153+0000 7f4130385700 1 -- 172.21.15.7:0/1629939650 >> 172.21.15.7:0/1629939650 conn(0x7f41280677e0 msgr2=0x7f41280a4420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:58.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.153+0000 7f4130385700 1 -- 172.21.15.7:0/1629939650 shutdown_connections 2023-09-30T01:59:58.158 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.153+0000 7f4130385700 1 -- 172.21.15.7:0/1629939650 wait complete. 2023-09-30T01:59:58.165 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-09-30T01:59:58.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.283+0000 7f383a97f700 10 monclient: get_monmap_and_config 2023-09-30T01:59:58.283 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.283+0000 7f383a97f700 10 monclient: build_initial_monmap 2023-09-30T01:59:58.284 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.283+0000 7f383a97f700 10 monclient: monmap: 2023-09-30T01:59:58.284 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:58.284 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:58.284 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:58.284742+0000 2023-09-30T01:59:58.284 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:58.284742+0000 2023-09-30T01:59:58.284 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:58.284 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:58.284 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:58.284 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:58.284 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:58.285 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:58.285 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:58.285 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:58.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.284+0000 7f383a97f700 1 Processor -- start 2023-09-30T01:59:58.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.284+0000 7f383a97f700 1 -- start start 2023-09-30T01:59:58.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.284+0000 7f383a97f700 10 monclient: init 2023-09-30T01:59:58.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.284+0000 7f383a97f700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:58.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.284+0000 7f383a97f700 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-09-30T01:59:58.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.284+0000 7f383a97f700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f3834098540 0x7f38341329e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.291 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.284+0000 7f383a97f700 10 monclient(hunting): picked mon.noname-c con 0x7f3834098540 addr v2:172.21.15.50:3300/0 2023-09-30T01:59:58.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.284+0000 7f383a97f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3834098910 addr v1:172.21.15.50:6789/0 2023-09-30T01:59:58.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.284+0000 7f383a97f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3834098d10 0x7f383412f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a97f700 10 monclient(hunting): picked mon.noname-a con 0x7f3834098d10 addr v2:172.21.15.7:3300/0 2023-09-30T01:59:58.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a97f700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3834069f10 con 0x7f3834098910 2023-09-30T01:59:58.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a97f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:58.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a97f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38340a0f40 con 0x7f3834098d10 2023-09-30T01:59:58.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a97f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:58.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a97f700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3834132f20 con 0x7f3834098540 2023-09-30T01:59:58.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a97f700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:58.292 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a97f700 10 monclient(hunting): authenticate will time out at 1487.350055s 2023-09-30T01:59:58.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a17e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3834098d10 0x7f383412f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a17e700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f3834098d10 0x7f383412f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49416/0 (socket says 172.21.15.7:49416) 2023-09-30T01:59:58.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a17e700 1 -- 172.21.15.7:0/1870327835 learned_addr learned my addr 172.21.15.7:0/1870327835 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:58.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a17e700 10 monclient(hunting): get_auth_request con 0x7f3834098d10 auth_method 0 2023-09-30T01:59:58.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a17e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:58.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a17e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:58.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a17e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:58.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a17e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:58.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a17e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:58.293 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a17e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:58.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383997d700 1 --2- 172.21.15.7:0/1870327835 >> v2:172.21.15.50:3300/0 conn(0x7f3834098540 0x7f38341329e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383997d700 10 monclient(hunting): get_auth_request con 0x7f3834098540 auth_method 0 2023-09-30T01:59:58.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383997d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:58.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383997d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:58.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383997d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:58.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a17e700 10 monclient(hunting): handle_auth_done global_id 4785 payload 274 2023-09-30T01:59:58.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a17e700 1 -- 172.21.15.7:0/1870327835 >> v2:172.21.15.50:3300/0 conn(0x7f3834098540 msgr2=0x7f38341329e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:58.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a17e700 1 --2- 172.21.15.7:0/1870327835 >> v2:172.21.15.50:3300/0 conn(0x7f3834098540 0x7f38341329e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a17e700 1 -- 172.21.15.7:0/1870327835 >> v1:172.21.15.50:6789/0 conn(0x7f3834098910 legacy=0x7f383412ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:58.294 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a17e700 10 monclient: _finish_hunting 0 2023-09-30T01:59:58.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a17e700 1 monclient: found mon.noname-a 2023-09-30T01:59:58.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a17e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:58.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a17e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:58.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.285+0000 7f383a17e700 1 -- 172.21.15.7:0/1870327835 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3834096a50 con 0x7f3834098d10 2023-09-30T01:59:58.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.287+0000 7f383a17e700 1 -- 172.21.15.7:0/1870327835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834098d10 msgr2=0x7f383412f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:58.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.287+0000 7f383a17e700 1 --2- 172.21.15.7:0/1870327835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834098d10 0x7f383412f2d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3834071cb0 tx=0x7f382c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb6c4fa51a148a60 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:58.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.287+0000 7f383897b700 1 -- 172.21.15.7:0/1870327835 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c00ba30 con 0x7f3834098d10 2023-09-30T01:59:58.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.287+0000 7f383897b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:58.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.287+0000 7f383897b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:58.295 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.287+0000 7f383897b700 10 monclient: dump: 2023-09-30T01:59:58.296 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:58.296 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:58.296 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.296 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.296 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:58.296 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:58.296 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:58.296 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:58.296 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:58.296 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:58.296 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.287+0000 7f383897b700 10 monclient: _finish_auth 0 2023-09-30T01:59:58.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.287+0000 7f383897b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:58.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.287+0000 7f383a97f700 5 monclient: authenticate success, global_id 4785 2023-09-30T01:59:58.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.287+0000 7f383897b700 1 -- 172.21.15.7:0/1870327835 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382c00bb90 con 0x7f3834098d10 2023-09-30T01:59:58.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.287+0000 7f383897b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:58.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.287+0000 7f383a97f700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:58.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.287+0000 7f383a97f700 4 set_mon_vals no callback set 2023-09-30T01:59:58.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.287+0000 7f383897b700 1 -- 172.21.15.7:0/1870327835 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c011420 con 0x7f3834098d10 2023-09-30T01:59:58.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.287+0000 7f383897b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:58.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.287+0000 7f383897b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:58.297 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.287+0000 7f383897b700 10 monclient: dump: 2023-09-30T01:59:58.297 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:58.298 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:58.298 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.298 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.298 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:58.298 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:58.298 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:58.298 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:58.298 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:58.298 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:58.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.288+0000 7f383a97f700 10 monclient: shutdown 2023-09-30T01:59:58.298 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.288+0000 7f383a97f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:58.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.288+0000 7f383a97f700 1 -- 172.21.15.7:0/1870327835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834098d10 msgr2=0x7f383412f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.288+0000 7f383a97f700 1 --2- 172.21.15.7:0/1870327835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834098d10 0x7f383412f2d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3834071cb0 tx=0x7f382c009250 comp rx=0 tx=0).stop 2023-09-30T01:59:58.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.288+0000 7f383a97f700 1 -- 172.21.15.7:0/1870327835 shutdown_connections 2023-09-30T01:59:58.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.288+0000 7f383a97f700 1 --2- 172.21.15.7:0/1870327835 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834098d10 0x7f383412f2d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.288+0000 7f383a97f700 1 --2- 172.21.15.7:0/1870327835 >> v2:172.21.15.50:3300/0 conn(0x7f3834098540 0x7f38341329e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.288+0000 7f383a97f700 1 -- 172.21.15.7:0/1870327835 >> 172.21.15.7:0/1870327835 conn(0x7f3834071830 msgr2=0x7f38340a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:58.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.288+0000 7f383a97f700 1 -- 172.21.15.7:0/1870327835 shutdown_connections 2023-09-30T01:59:58.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.288+0000 7f383a97f700 1 -- 172.21.15.7:0/1870327835 wait complete. 2023-09-30T01:59:58.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.288+0000 7f383a97f700 10 monclient: build_initial_monmap 2023-09-30T01:59:58.299 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.288+0000 7f383a97f700 10 monclient: monmap: 2023-09-30T01:59:58.299 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:58.299 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:58.300 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:58.300 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:58.300 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:58.300 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:58.300 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:58.300 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:58.300 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:58.300 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:58.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.289+0000 7f383a97f700 1 Processor -- start 2023-09-30T01:59:58.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.289+0000 7f383a97f700 1 -- start start 2023-09-30T01:59:58.300 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.289+0000 7f383a97f700 10 monclient: init 2023-09-30T01:59:58.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.289+0000 7f383a97f700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:58.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.289+0000 7f383a97f700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:59:58.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.289+0000 7f383a97f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3834098540 0x7f3834141680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.289+0000 7f383a97f700 10 monclient(hunting): picked mon.noname-b con 0x7f3834098540 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:58.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.289+0000 7f383a97f700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3834098910 0x7f383412ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.289+0000 7f383a97f700 10 monclient(hunting): picked mon.noname-c con 0x7f3834098910 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:58.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.289+0000 7f383a97f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834098d10 0x7f3834099300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.289+0000 7f383a97f700 10 monclient(hunting): picked mon.noname-a con 0x7f3834098d10 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:58.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.289+0000 7f383a97f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:58.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.289+0000 7f383a97f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383409b2a0 con 0x7f3834098d10 2023-09-30T01:59:58.301 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.289+0000 7f383a97f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:58.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.289+0000 7f383a97f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38341327b0 con 0x7f3834098540 2023-09-30T01:59:58.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.289+0000 7f383a97f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:58.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.289+0000 7f383a97f700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38340a3dc0 con 0x7f3834098910 2023-09-30T01:59:58.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.289+0000 7f383a97f700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:58.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.289+0000 7f383a97f700 10 monclient(hunting): authenticate will time out at 1487.354883s 2023-09-30T01:59:58.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.289+0000 7f383917c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3834098910 0x7f383412ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.289+0000 7f383a17e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834098d10 0x7f3834099300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383997d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3834098540 0x7f3834141680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383a17e700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834098d10 0x7f3834099300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49428/0 (socket says 172.21.15.7:49428) 2023-09-30T01:59:58.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383a17e700 1 -- 172.21.15.7:0/4003560864 learned_addr learned my addr 172.21.15.7:0/4003560864 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:58.302 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383a17e700 10 monclient(hunting): get_auth_request con 0x7f3834098d10 auth_method 0 2023-09-30T01:59:58.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383a17e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:58.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383a17e700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:58.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383a17e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:58.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383917c700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3834098910 0x7f383412ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41436/0 (socket says 172.21.15.7:41436) 2023-09-30T01:59:58.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383917c700 10 monclient(hunting): get_auth_request con 0x7f3834098910 auth_method 0 2023-09-30T01:59:58.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383917c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:58.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383917c700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:58.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383917c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:58.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383997d700 10 monclient(hunting): get_auth_request con 0x7f3834098540 auth_method 0 2023-09-30T01:59:58.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383997d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:58.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383997d700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:58.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383997d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:58.303 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383a17e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:58.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383a17e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:58.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383a17e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:58.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383917c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:58.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383917c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:58.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383917c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:58.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383997d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:58.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383997d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:58.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383997d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:58.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383a17e700 10 monclient(hunting): handle_auth_done global_id 4788 payload 824 2023-09-30T01:59:58.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383a17e700 1 -- 172.21.15.7:0/4003560864 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3834098910 msgr2=0x7f383412ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.304 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383a17e700 1 --2- 172.21.15.7:0/4003560864 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3834098910 0x7f383412ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383a17e700 1 -- 172.21.15.7:0/4003560864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3834098540 msgr2=0x7f3834141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383a17e700 1 --2- 172.21.15.7:0/4003560864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3834098540 0x7f3834141680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383a17e700 10 monclient: _finish_hunting 0 2023-09-30T01:59:58.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383a17e700 1 monclient: found mon.noname-a 2023-09-30T01:59:58.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383a17e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:58.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383a17e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:58.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383a17e700 1 -- 172.21.15.7:0/4003560864 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f382c0094f0 con 0x7f3834098d10 2023-09-30T01:59:58.305 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.290+0000 7f383917c700 1 --2- 172.21.15.7:0/4003560864 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3834098910 0x7f383412ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:58.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.292+0000 7f383a17e700 1 -- 172.21.15.7:0/4003560864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834098d10 msgr2=0x7f3834099300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:58.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.292+0000 7f383a17e700 1 --2- 172.21.15.7:0/4003560864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834098d10 0x7f3834099300 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f382c00b3c0 tx=0x7f382c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:58.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.292+0000 7f3831ffb700 1 -- 172.21.15.7:0/4003560864 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c011e40 con 0x7f3834098d10 2023-09-30T01:59:58.307 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.292+0000 7f3831ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:58.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.292+0000 7f3831ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:58.308 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.292+0000 7f3831ffb700 10 monclient: dump: 2023-09-30T01:59:58.308 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:58.308 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:58.308 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.308 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.308 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:58.308 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:58.308 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:58.308 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:58.309 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:58.309 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:58.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.292+0000 7f3831ffb700 10 monclient: _finish_auth 0 2023-09-30T01:59:58.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.292+0000 7f3831ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:58.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.292+0000 7f3831ffb700 1 -- 172.21.15.7:0/4003560864 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382c020070 con 0x7f3834098d10 2023-09-30T01:59:58.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.292+0000 7f383a97f700 5 monclient: authenticate success, global_id 4788 2023-09-30T01:59:58.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.292+0000 7f3831ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:58.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.292+0000 7f3831ffb700 1 -- 172.21.15.7:0/4003560864 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c01c6a0 con 0x7f3834098d10 2023-09-30T01:59:58.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.292+0000 7f383a97f700 10 monclient: _renew_subs 2023-09-30T01:59:58.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.292+0000 7f383a97f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:58.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.292+0000 7f383a97f700 1 -- 172.21.15.7:0/4003560864 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38340a05c0 con 0x7f3834098d10 2023-09-30T01:59:58.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.292+0000 7f3833fff700 4 set_mon_vals no callback set 2023-09-30T01:59:58.309 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.292+0000 7f3831ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:58.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.292+0000 7f3831ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:58.310 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.292+0000 7f3831ffb700 10 monclient: dump: 2023-09-30T01:59:58.310 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:58.310 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:58.310 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.310 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.310 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:58.310 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:58.310 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:58.310 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:58.310 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:58.310 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:58.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.292+0000 7f383a97f700 10 monclient: _renew_subs 2023-09-30T01:59:58.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.293+0000 7f383a97f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:58.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.293+0000 7f383a97f700 1 -- 172.21.15.7:0/4003560864 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f383409b960 con 0x7f3834098d10 2023-09-30T01:59:58.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.293+0000 7f3831ffb700 1 -- 172.21.15.7:0/4003560864 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f382c022030 con 0x7f3834098d10 2023-09-30T01:59:58.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.293+0000 7f383a97f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:58.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.293+0000 7f383a97f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:58.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.293+0000 7f383a97f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:58.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.293+0000 7f3831ffb700 1 --2- 172.21.15.7:0/4003560864 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3820035860 0x7f3820037d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.293+0000 7f383a97f700 1 -- 172.21.15.7:0/4003560864 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3834064c10 con 0x7f3834098d10 2023-09-30T01:59:58.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.294+0000 7f3831ffb700 1 -- 172.21.15.7:0/4003560864 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f382c049ed0 con 0x7f3834098d10 2023-09-30T01:59:58.311 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.294+0000 7f383997d700 1 --2- 172.21.15.7:0/4003560864 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3820035860 0x7f3820037d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.294+0000 7f383997d700 10 monclient: get_auth_request con 0x7f3820035860 auth_method 0 2023-09-30T01:59:58.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.296+0000 7f383997d700 1 --2- 172.21.15.7:0/4003560864 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3820035860 0x7f3820037d10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f38240075a0 tx=0x7f3824009880 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:58.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.298+0000 7f3831ffb700 1 -- 172.21.15.7:0/4003560864 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f382c010790 con 0x7f3834098d10 2023-09-30T01:59:58.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.298+0000 7f3831ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:58.312 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.298+0000 7f3831ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:58.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.494+0000 7f383a97f700 1 -- 172.21.15.7:0/4003560864 --> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f3834064c10 con 0x7f3820035860 2023-09-30T01:59:58.494 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.495+0000 7f3831ffb700 1 -- 172.21.15.7:0/4003560864 <== mgr.4105 v2:172.21.15.50:6824/86200 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3834064c10 con 0x7f3820035860 2023-09-30T01:59:58.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.497+0000 7f383a97f700 1 -- 172.21.15.7:0/4003560864 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3820035860 msgr2=0x7f3820037d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.497 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.497+0000 7f383a97f700 1 --2- 172.21.15.7:0/4003560864 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3820035860 0x7f3820037d10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f38240075a0 tx=0x7f3824009880 comp rx=0 tx=0).stop 2023-09-30T01:59:58.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.497+0000 7f383a97f700 10 monclient: shutdown 2023-09-30T01:59:58.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.497+0000 7f383a97f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:58.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.497+0000 7f383a97f700 1 -- 172.21.15.7:0/4003560864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834098d10 msgr2=0x7f3834099300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.497+0000 7f383a97f700 1 --2- 172.21.15.7:0/4003560864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834098d10 0x7f3834099300 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f382c00b3c0 tx=0x7f382c016000 comp rx=0 tx=0).stop 2023-09-30T01:59:58.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.497+0000 7f383a97f700 1 -- 172.21.15.7:0/4003560864 shutdown_connections 2023-09-30T01:59:58.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.497+0000 7f383a97f700 1 --2- 172.21.15.7:0/4003560864 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f3820035860 0x7f3820037d10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.497+0000 7f383a97f700 1 --2- 172.21.15.7:0/4003560864 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3834098d10 0x7f3834099300 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.497+0000 7f383a97f700 1 --2- 172.21.15.7:0/4003560864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3834098540 0x7f3834141680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.497+0000 7f383a97f700 1 --2- 172.21.15.7:0/4003560864 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f3834098910 0x7f383412ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.498 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.497+0000 7f383a97f700 1 -- 172.21.15.7:0/4003560864 >> 172.21.15.7:0/4003560864 conn(0x7f3834071830 msgr2=0x7f38340a5570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:58.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.497+0000 7f383a97f700 1 -- 172.21.15.7:0/4003560864 shutdown_connections 2023-09-30T01:59:58.499 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.497+0000 7f383a97f700 1 -- 172.21.15.7:0/4003560864 wait complete. 2023-09-30T01:59:58.499 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 3 to deep-scrub 2023-09-30T01:59:58.508 INFO:tasks.ceph:Scrubbing osd.4 2023-09-30T01:59:58.509 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:59:58.630 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.630+0000 7f92c4c5f700 10 monclient: get_monmap_and_config 2023-09-30T01:59:58.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.630+0000 7f92c4c5f700 10 monclient: build_initial_monmap 2023-09-30T01:59:58.631 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.630+0000 7f92c4c5f700 10 monclient: monmap: 2023-09-30T01:59:58.631 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:58.631 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:58.631 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:58.631800+0000 2023-09-30T01:59:58.631 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:58.631800+0000 2023-09-30T01:59:58.631 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:58.631 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:58.631 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:58.632 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:58.632 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:58.632 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:58.632 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:58.632 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:58.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.631+0000 7f92c4c5f700 1 Processor -- start 2023-09-30T01:59:58.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.631+0000 7f92c4c5f700 1 -- start start 2023-09-30T01:59:58.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.631+0000 7f92c4c5f700 10 monclient: init 2023-09-30T01:59:58.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.631+0000 7f92c4c5f700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:58.633 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.631+0000 7f92c4c5f700 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-09-30T01:59:58.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.631+0000 7f92c4c5f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f92c00985c0 addr v1:172.21.15.7:6789/0 2023-09-30T01:59:58.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.631+0000 7f92c4c5f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f92c0098990 addr v1:172.21.15.50:6789/0 2023-09-30T01:59:58.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.631+0000 7f92c4c5f700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f92c0098da0 0x7f92c013a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.631+0000 7f92c4c5f700 10 monclient(hunting): picked mon.noname-a con 0x7f92c0098da0 addr v2:172.21.15.7:3300/0 2023-09-30T01:59:58.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.631+0000 7f92c4c5f700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c0132da0 con 0x7f92c00985c0 2023-09-30T01:59:58.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.632+0000 7f92c4c5f700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c00a1c90 con 0x7f92c0098990 2023-09-30T01:59:58.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.632+0000 7f92c4c5f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:58.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.632+0000 7f92c4c5f700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c00a0800 con 0x7f92c0098da0 2023-09-30T01:59:58.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.632+0000 7f92c4c5f700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:58.634 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.632+0000 7f92c4c5f700 10 monclient(hunting): authenticate will time out at 1487.697017s 2023-09-30T01:59:58.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.632+0000 7f92bffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f92c0098da0 0x7f92c013a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.632+0000 7f92bffff700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f92c0098da0 0x7f92c013a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49434/0 (socket says 172.21.15.7:49434) 2023-09-30T01:59:58.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.632+0000 7f92bffff700 1 -- 172.21.15.7:0/145566262 learned_addr learned my addr 172.21.15.7:0/145566262 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:58.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.632+0000 7f92bffff700 10 monclient(hunting): get_auth_request con 0x7f92c0098da0 auth_method 0 2023-09-30T01:59:58.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.632+0000 7f92bffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:58.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.632+0000 7f92bffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:58.635 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.632+0000 7f92bffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:58.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.634+0000 7f92bf7fe700 1 -- 172.21.15.7:0/145566262 >> v1:172.21.15.7:6789/0 conn(0x7f92c00985c0 legacy=0x7f92c0132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:58.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.634+0000 7f92bffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:58.640 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.634+0000 7f92bffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:58.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.634+0000 7f92bffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:58.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.634+0000 7f92bffff700 10 monclient(hunting): handle_auth_done global_id 4791 payload 274 2023-09-30T01:59:58.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.634+0000 7f92bffff700 1 -- 172.21.15.7:0/145566262 >> v1:172.21.15.50:6789/0 conn(0x7f92c0098990 legacy=0x7f92c01372a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:58.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.634+0000 7f92bffff700 1 -- 172.21.15.7:0/145566262 >> v1:172.21.15.7:6789/0 conn(0x7f92c00985c0 legacy=0x7f92c0132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.634+0000 7f92bffff700 10 monclient: _finish_hunting 0 2023-09-30T01:59:58.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.634+0000 7f92bffff700 1 monclient: found mon.noname-a 2023-09-30T01:59:58.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.634+0000 7f92bffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:58.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.634+0000 7f92bffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:58.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.634+0000 7f92beffd700 1 -- 172.21.15.7:0/145566262 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 456924239 0 0) 0x7f92c0132da0 con 0x7f92c00985c0 2023-09-30T01:59:58.641 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.634+0000 7f92bffff700 1 -- 172.21.15.7:0/145566262 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c00a19b0 con 0x7f92c0098da0 2023-09-30T01:59:58.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.634+0000 7f92beffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T01:59:58.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.636+0000 7f92bffff700 1 -- 172.21.15.7:0/145566262 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92c0098da0 msgr2=0x7f92c013a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:58.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.636+0000 7f92bffff700 1 --2- 172.21.15.7:0/145566262 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92c0098da0 0x7f92c013a9d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f92b8009fa0 tx=0x7f92b8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e20c75e5b815a26 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:58.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.636+0000 7f92beffd700 1 -- 172.21.15.7:0/145566262 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b800e070 con 0x7f92c0098da0 2023-09-30T01:59:58.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:58.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92beffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:58.642 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92beffd700 10 monclient: dump: 2023-09-30T01:59:58.642 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:58.642 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:58.642 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.642 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.642 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:58.643 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:58.643 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:58.643 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:58.643 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:58.643 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:58.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92beffd700 10 monclient: _finish_auth 0 2023-09-30T01:59:58.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92beffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:58.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92c4c5f700 5 monclient: authenticate success, global_id 4791 2023-09-30T01:59:58.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92beffd700 1 -- 172.21.15.7:0/145566262 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b800bcd0 con 0x7f92c0098da0 2023-09-30T01:59:58.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92beffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:58.643 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92c4c5f700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:58.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92beffd700 1 -- 172.21.15.7:0/145566262 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b8012680 con 0x7f92c0098da0 2023-09-30T01:59:58.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:58.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92c4c5f700 4 set_mon_vals no callback set 2023-09-30T01:59:58.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92beffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:58.644 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92beffd700 10 monclient: dump: 2023-09-30T01:59:58.644 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:58.644 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:58.644 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.644 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.644 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:58.644 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:58.645 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:58.645 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:58.645 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:58.645 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:58.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92c4c5f700 10 monclient: shutdown 2023-09-30T01:59:58.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92c4c5f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:58.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92c4c5f700 1 -- 172.21.15.7:0/145566262 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92c0098da0 msgr2=0x7f92c013a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92c4c5f700 1 --2- 172.21.15.7:0/145566262 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92c0098da0 0x7f92c013a9d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f92b8009fa0 tx=0x7f92b8009840 comp rx=0 tx=0).stop 2023-09-30T01:59:58.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92c4c5f700 1 -- 172.21.15.7:0/145566262 shutdown_connections 2023-09-30T01:59:58.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92c4c5f700 1 --2- 172.21.15.7:0/145566262 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92c0098da0 0x7f92c013a9d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.645 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92c4c5f700 1 -- 172.21.15.7:0/145566262 >> 172.21.15.7:0/145566262 conn(0x7f92c00a2070 msgr2=0x7f92c00a2470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:58.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92c4c5f700 1 -- 172.21.15.7:0/145566262 shutdown_connections 2023-09-30T01:59:58.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.637+0000 7f92c4c5f700 1 -- 172.21.15.7:0/145566262 wait complete. 2023-09-30T01:59:58.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.638+0000 7f92c4c5f700 10 monclient: build_initial_monmap 2023-09-30T01:59:58.646 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.638+0000 7f92c4c5f700 10 monclient: monmap: 2023-09-30T01:59:58.646 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:58.646 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:58.646 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:58.646 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:58.646 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:58.646 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:58.647 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:58.647 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:58.647 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:58.647 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:58.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.638+0000 7f92c4c5f700 1 Processor -- start 2023-09-30T01:59:58.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.638+0000 7f92c4c5f700 1 -- start start 2023-09-30T01:59:58.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.638+0000 7f92c4c5f700 10 monclient: init 2023-09-30T01:59:58.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.638+0000 7f92c4c5f700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:58.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.638+0000 7f92c4c5f700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T01:59:58.647 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.638+0000 7f92c4c5f700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92c00985c0 0x7f92c0141400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.638+0000 7f92c4c5f700 10 monclient(hunting): picked mon.noname-a con 0x7f92c00985c0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:58.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.638+0000 7f92c4c5f700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92c0098990 0x7f92c012f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.638+0000 7f92c4c5f700 10 monclient(hunting): picked mon.noname-c con 0x7f92c0098990 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:58.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.638+0000 7f92c4c5f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92c0098da0 0x7f92c0099600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.638+0000 7f92c4c5f700 10 monclient(hunting): picked mon.noname-b con 0x7f92c0098da0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:58.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.638+0000 7f92c4c5f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:58.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92c4c5f700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c00a3a50 con 0x7f92c00985c0 2023-09-30T01:59:58.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92c4c5f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:58.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92c4c5f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c011c310 con 0x7f92c0098da0 2023-09-30T01:59:58.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92c4c5f700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:58.648 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92c4c5f700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c011c5f0 con 0x7f92c0098990 2023-09-30T01:59:58.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92c4c5f700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:58.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92c4c5f700 10 monclient(hunting): authenticate will time out at 1487.704019s 2023-09-30T01:59:58.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92bf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92c00985c0 0x7f92c0141400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92b6ffd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92c0098990 0x7f92c012f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92bf7fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92c00985c0 0x7f92c0141400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49450/0 (socket says 172.21.15.7:49450) 2023-09-30T01:59:58.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92b6ffd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92c0098990 0x7f92c012f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41450/0 (socket says 172.21.15.7:41450) 2023-09-30T01:59:58.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92bf7fe700 1 -- 172.21.15.7:0/3743457859 learned_addr learned my addr 172.21.15.7:0/3743457859 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:58.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92bf7fe700 10 monclient(hunting): get_auth_request con 0x7f92c00985c0 auth_method 0 2023-09-30T01:59:58.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92bf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:58.649 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92bf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:58.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92bf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:58.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92b6ffd700 10 monclient(hunting): get_auth_request con 0x7f92c0098990 auth_method 0 2023-09-30T01:59:58.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92b6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:58.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92b6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:58.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92b6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:58.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92b6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:58.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92b6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:58.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92b6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:58.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92bf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:58.650 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92bf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92bf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92bffff700 1 --2- 172.21.15.7:0/3743457859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92c0098da0 0x7f92c0099600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92b6ffd700 10 monclient(hunting): handle_auth_done global_id 4817 payload 824 2023-09-30T01:59:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92b6ffd700 1 -- 172.21.15.7:0/3743457859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92c0098da0 msgr2=0x7f92c0099600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92b6ffd700 1 --2- 172.21.15.7:0/3743457859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92c0098da0 0x7f92c0099600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92b6ffd700 1 -- 172.21.15.7:0/3743457859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92c00985c0 msgr2=0x7f92c0141400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92b6ffd700 1 --2- 172.21.15.7:0/3743457859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f92c00985c0 0x7f92c0141400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92b6ffd700 10 monclient: _finish_hunting 0 2023-09-30T01:59:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92b6ffd700 1 monclient: found mon.noname-c 2023-09-30T01:59:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92b6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:58.651 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92b6ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:59:58.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92b6ffd700 1 -- 172.21.15.7:0/3743457859 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b8009ae0 con 0x7f92c0098990 2023-09-30T01:59:58.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92bffff700 10 monclient: get_auth_request con 0x7f92c0098da0 auth_method 0 2023-09-30T01:59:58.652 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.639+0000 7f92bffff700 1 --2- 172.21.15.7:0/3743457859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92c0098da0 0x7f92c0099600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:59:58.653 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92b6ffd700 1 -- 172.21.15.7:0/3743457859 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92c0098990 msgr2=0x7f92c012f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:58.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92b6ffd700 1 --2- 172.21.15.7:0/3743457859 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92c0098990 0x7f92c012f1f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f92a800c4e0 tx=0x7f92a8007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:58.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92b7fff700 1 -- 172.21.15.7:0/3743457859 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a80117a0 con 0x7f92c0098990 2023-09-30T01:59:58.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92b7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:58.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92b7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:59:58.654 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92b7fff700 10 monclient: dump: 2023-09-30T01:59:58.654 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:58.654 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:58.654 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.654 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.654 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:58.655 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:58.655 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:58.655 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:58.655 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:58.655 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:58.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92b7fff700 10 monclient: _finish_auth 0 2023-09-30T01:59:58.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92b7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:58.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92c4c5f700 5 monclient: authenticate success, global_id 4817 2023-09-30T01:59:58.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92c4c5f700 10 monclient: _renew_subs 2023-09-30T01:59:58.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92b7fff700 1 -- 172.21.15.7:0/3743457859 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a8011920 con 0x7f92c0098990 2023-09-30T01:59:58.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92c4c5f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:58.655 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92c4c5f700 1 -- 172.21.15.7:0/3743457859 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c009b2a0 con 0x7f92c0098990 2023-09-30T01:59:58.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92b7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:58.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92b7fff700 1 -- 172.21.15.7:0/3743457859 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a8011ce0 con 0x7f92c0098990 2023-09-30T01:59:58.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92b7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:58.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92b7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:58.656 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92b7fff700 10 monclient: dump: 2023-09-30T01:59:58.656 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:58.656 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:58.656 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.656 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:58.656 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:58.656 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:58.656 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:58.657 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:58.657 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:58.657 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:58.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92c4c5f700 10 monclient: _renew_subs 2023-09-30T01:59:58.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92be7fc700 4 set_mon_vals no callback set 2023-09-30T01:59:58.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92c4c5f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:58.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.641+0000 7f92c4c5f700 1 -- 172.21.15.7:0/3743457859 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c01420f0 con 0x7f92c0098990 2023-09-30T01:59:58.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.642+0000 7f92c4c5f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:58.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.642+0000 7f92c4c5f700 1 -- 172.21.15.7:0/3743457859 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f92c0142310 con 0x7f92c0098990 2023-09-30T01:59:58.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.642+0000 7f92b6ffd700 1 --2- 172.21.15.7:0/3743457859 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92c0098990 0x7f92c012f1f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f92a800c4e0 tx=0x7f92a8007ea0 comp rx=0 tx=0).queue_received will delay after 1.69604e+09 on 0x7f92a80479a0 osd_map(15..15 src has 1..15) v4 2023-09-30T01:59:58.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.642+0000 7f92b7fff700 1 -- 172.21.15.7:0/3743457859 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f92a8012630 con 0x7f92c0098990 2023-09-30T01:59:58.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.642+0000 7f92b7fff700 1 --2- 172.21.15.7:0/3743457859 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f92a00357e0 0x7f92a0037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.657 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.643+0000 7f92bf7fe700 1 --2- 172.21.15.7:0/3743457859 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f92a00357e0 0x7f92a0037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.643+0000 7f92bf7fe700 10 monclient: get_auth_request con 0x7f92a00357e0 auth_method 0 2023-09-30T01:59:58.658 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.645+0000 7f92bf7fe700 1 --2- 172.21.15.7:0/3743457859 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f92a00357e0 0x7f92a0037c90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f92b0006ee0 tx=0x7f92b0007c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:58.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.867+0000 7f92b7fff700 1 -- 172.21.15.7:0/3743457859 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f92a80479a0 con 0x7f92c0098990 2023-09-30T01:59:58.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.867+0000 7f92b7fff700 1 --2- 172.21.15.7:0/3743457859 >> [v2:172.21.15.50:6803/86632,v1:172.21.15.50:6806/86632] conn(0x7f92a003b5e0 0x7f92a003d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:58.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.867+0000 7f92b7fff700 1 -- 172.21.15.7:0/3743457859 --> [v2:172.21.15.50:6803/86632,v1:172.21.15.50:6806/86632] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92a003e0a0 con 0x7f92a003b5e0 2023-09-30T01:59:58.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.867+0000 7f92bffff700 1 --2- 172.21.15.7:0/3743457859 >> [v2:172.21.15.50:6803/86632,v1:172.21.15.50:6806/86632] conn(0x7f92a003b5e0 0x7f92a003d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:58.868 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.867+0000 7f92bffff700 10 monclient: get_auth_request con 0x7f92a003b5e0 auth_method 0 2023-09-30T01:59:58.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.872+0000 7f92bffff700 1 --2- 172.21.15.7:0/3743457859 >> [v2:172.21.15.50:6803/86632,v1:172.21.15.50:6806/86632] conn(0x7f92a003b5e0 0x7f92a003d9f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:58.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.873+0000 7f92b7fff700 1 -- 172.21.15.7:0/3743457859 <== osd.4 v2:172.21.15.50:6803/86632 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f92a003e0a0 con 0x7f92a003b5e0 2023-09-30T01:59:58.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.887+0000 7f92c4c5f700 1 -- 172.21.15.7:0/3743457859 --> [v2:172.21.15.50:6803/86632,v1:172.21.15.50:6806/86632] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f92c005f230 con 0x7f92a003b5e0 2023-09-30T01:59:58.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.889+0000 7f92b7fff700 1 -- 172.21.15.7:0/3743457859 <== osd.4 v2:172.21.15.50:6803/86632 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f92c005f230 con 0x7f92a003b5e0 2023-09-30T01:59:58.888 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-09-30T01:59:58.888 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:59:58.888 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-09-30T01:59:58.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.889+0000 7f92c4c5f700 1 -- 172.21.15.7:0/3743457859 >> [v2:172.21.15.50:6803/86632,v1:172.21.15.50:6806/86632] conn(0x7f92a003b5e0 msgr2=0x7f92a003d9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.889+0000 7f92c4c5f700 1 --2- 172.21.15.7:0/3743457859 >> [v2:172.21.15.50:6803/86632,v1:172.21.15.50:6806/86632] conn(0x7f92a003b5e0 0x7f92a003d9f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.889+0000 7f92c4c5f700 1 -- 172.21.15.7:0/3743457859 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f92a00357e0 msgr2=0x7f92a0037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.889+0000 7f92c4c5f700 1 --2- 172.21.15.7:0/3743457859 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f92a00357e0 0x7f92a0037c90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f92b0006ee0 tx=0x7f92b0007c00 comp rx=0 tx=0).stop 2023-09-30T01:59:58.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.889+0000 7f92c4c5f700 10 monclient: shutdown 2023-09-30T01:59:58.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.889+0000 7f92c4c5f700 20 monclient: shutdown canceling and discarding version request 1 2023-09-30T01:59:58.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.889+0000 7f92c4c5f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:58.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.889+0000 7f92c4c5f700 1 -- 172.21.15.7:0/3743457859 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92c0098990 msgr2=0x7f92c012f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:58.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.889+0000 7f92c4c5f700 1 --2- 172.21.15.7:0/3743457859 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f92c0098990 0x7f92c012f1f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f92a800c4e0 tx=0x7f92a8007ea0 comp rx=0 tx=0).stop 2023-09-30T01:59:58.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.889+0000 7f92bffff700 1 -- 172.21.15.7:0/3743457859 reap_dead start 2023-09-30T01:59:58.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.889+0000 7f92b7fff700 1 -- 172.21.15.7:0/3743457859 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (unknown 0 0 0) 0x7f92a80128e0 con 0x7f92c0098990 2023-09-30T01:59:58.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.889+0000 7f92b7fff700 10 monclient: discarding stray monitor message mon_get_version_reply(handle=1 version=15) v2 2023-09-30T01:59:58.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.889+0000 7f92c4c5f700 1 -- 172.21.15.7:0/3743457859 shutdown_connections 2023-09-30T01:59:58.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.889+0000 7f92c4c5f700 1 -- 172.21.15.7:0/3743457859 >> 172.21.15.7:0/3743457859 conn(0x7f92c00a2070 msgr2=0x7f92c0097310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:58.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.889+0000 7f92c4c5f700 1 -- 172.21.15.7:0/3743457859 shutdown_connections 2023-09-30T01:59:58.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:58.889+0000 7f92c4c5f700 1 -- 172.21.15.7:0/3743457859 wait complete. 2023-09-30T01:59:58.900 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-09-30T01:59:59.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.019+0000 7f7271ca6700 10 monclient: get_monmap_and_config 2023-09-30T01:59:59.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.019+0000 7f7271ca6700 10 monclient: build_initial_monmap 2023-09-30T01:59:59.019 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.019+0000 7f7271ca6700 10 monclient: monmap: 2023-09-30T01:59:59.020 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:59.020 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:59.020 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:59.020633+0000 2023-09-30T01:59:59.020 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:59.020633+0000 2023-09-30T01:59:59.020 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:59.020 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.020 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:59.020 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:59.020 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:59.020 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:59.021 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:59.021 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f7271ca6700 1 Processor -- start 2023-09-30T01:59:59.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f7271ca6700 1 -- start start 2023-09-30T01:59:59.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f7271ca6700 10 monclient: init 2023-09-30T01:59:59.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f7271ca6700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:59.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f7271ca6700 10 monclient: _add_conns ranks=[2,0,4,1,3] 2023-09-30T01:59:59.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f7271ca6700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f726c096a50 0x7f726c132680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f7271ca6700 10 monclient(hunting): picked mon.noname-b con 0x7f726c096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:59.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f7271ca6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f726c096e20 addr v1:172.21.15.7:6789/0 2023-09-30T01:59:59.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f7271ca6700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f726c137710 0x7f726c139af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.025 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f7271ca6700 10 monclient(hunting): picked mon.noname-c con 0x7f726c137710 addr v2:172.21.15.50:3300/0 2023-09-30T01:59:59.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f7271ca6700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f726c099440 con 0x7f726c096e20 2023-09-30T01:59:59.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f7271ca6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:59.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f7271ca6700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f726c097b40 con 0x7f726c137710 2023-09-30T01:59:59.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f7271ca6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:59.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f7271ca6700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f726c099160 con 0x7f726c096a50 2023-09-30T01:59:59.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f7271ca6700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:59.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f7271ca6700 10 monclient(hunting): authenticate will time out at 1488.085782s 2023-09-30T01:59:59.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f726affd700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f726c096e20 0x7f726c0971f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48970/0 (socket says 172.21.15.7:48970) 2023-09-30T01:59:59.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f726affd700 1 -- 172.21.15.7:0/153742073 learned_addr learned my addr 172.21.15.7:0/153742073 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:59:59.026 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f726b7fe700 1 --2- 172.21.15.7:0/153742073 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f726c096a50 0x7f726c132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f726b7fe700 10 monclient(hunting): get_auth_request con 0x7f726c096a50 auth_method 0 2023-09-30T01:59:59.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f726b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:59.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f726b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:59.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.020+0000 7f726b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:59.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.021+0000 7f726b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:59.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.021+0000 7f726b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:59.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.021+0000 7f726bfff700 1 --2- 172.21.15.7:0/153742073 >> v2:172.21.15.50:3300/0 conn(0x7f726c137710 0x7f726c139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.021+0000 7f726b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:59.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.021+0000 7f726bfff700 10 monclient(hunting): get_auth_request con 0x7f726c137710 auth_method 0 2023-09-30T01:59:59.027 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.021+0000 7f726bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:59.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.021+0000 7f726bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:59.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.021+0000 7f726bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:59.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.021+0000 7f726bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:59.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.021+0000 7f726bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:59.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.021+0000 7f726bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:59.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.021+0000 7f726b7fe700 10 monclient(hunting): handle_auth_done global_id 4820 payload 274 2023-09-30T01:59:59.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.021+0000 7f726b7fe700 1 -- 172.21.15.7:0/153742073 >> v2:172.21.15.50:3300/0 conn(0x7f726c137710 msgr2=0x7f726c139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:59.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.021+0000 7f726b7fe700 1 --2- 172.21.15.7:0/153742073 >> v2:172.21.15.50:3300/0 conn(0x7f726c137710 0x7f726c139af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.021+0000 7f726b7fe700 1 -- 172.21.15.7:0/153742073 >> v1:172.21.15.7:6789/0 conn(0x7f726c096e20 legacy=0x7f726c0971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:59.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.021+0000 7f726b7fe700 10 monclient: _finish_hunting 0 2023-09-30T01:59:59.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.021+0000 7f726b7fe700 1 monclient: found mon.noname-b 2023-09-30T01:59:59.028 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.021+0000 7f726b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:59.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.021+0000 7f726b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:59.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.021+0000 7f726b7fe700 1 -- 172.21.15.7:0/153742073 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726c099720 con 0x7f726c096a50 2023-09-30T01:59:59.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.023+0000 7f726b7fe700 1 -- 172.21.15.7:0/153742073 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f726c096a50 msgr2=0x7f726c132680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:59.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.023+0000 7f726b7fe700 1 --2- 172.21.15.7:0/153742073 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f726c096a50 0x7f726c132680 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f725c0098e0 tx=0x7f725c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bca8943a4c97971e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:59.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.023+0000 7f726a7fc700 1 -- 172.21.15.7:0/153742073 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c011840 con 0x7f726c096a50 2023-09-30T01:59:59.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.023+0000 7f726a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:59.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.023+0000 7f726a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:59.029 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.023+0000 7f726a7fc700 10 monclient: dump: 2023-09-30T01:59:59.029 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:59.029 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:59.029 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.029 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.030 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:59.030 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.030 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:59.030 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:59.030 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:59.030 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.023+0000 7f726a7fc700 10 monclient: _finish_auth 0 2023-09-30T01:59:59.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.023+0000 7f726a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:59.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.023+0000 7f726a7fc700 1 -- 172.21.15.7:0/153742073 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725c0119a0 con 0x7f726c096a50 2023-09-30T01:59:59.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.023+0000 7f7271ca6700 5 monclient: authenticate success, global_id 4820 2023-09-30T01:59:59.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.023+0000 7f7271ca6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:59.030 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.023+0000 7f726a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:59.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.023+0000 7f7271ca6700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:59.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.023+0000 7f726a7fc700 1 -- 172.21.15.7:0/153742073 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c011d40 con 0x7f726c096a50 2023-09-30T01:59:59.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.023+0000 7f7271ca6700 4 set_mon_vals no callback set 2023-09-30T01:59:59.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.023+0000 7f726a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:59.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.023+0000 7f726a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:59.031 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.023+0000 7f726a7fc700 10 monclient: dump: 2023-09-30T01:59:59.031 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:59.031 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:59.031 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.031 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.031 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:59.032 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.032 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:59.032 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:59.032 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:59.032 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.024+0000 7f7271ca6700 10 monclient: shutdown 2023-09-30T01:59:59.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.024+0000 7f7271ca6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:59.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.024+0000 7f7271ca6700 1 -- 172.21.15.7:0/153742073 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f726c096a50 msgr2=0x7f726c132680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:59.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.024+0000 7f7271ca6700 1 --2- 172.21.15.7:0/153742073 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f726c096a50 0x7f726c132680 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f725c0098e0 tx=0x7f725c0091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.024+0000 7f7271ca6700 1 -- 172.21.15.7:0/153742073 shutdown_connections 2023-09-30T01:59:59.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.024+0000 7f7271ca6700 1 --2- 172.21.15.7:0/153742073 >> v2:172.21.15.50:3300/0 conn(0x7f726c137710 0x7f726c139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.032 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.024+0000 7f7271ca6700 1 --2- 172.21.15.7:0/153742073 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f726c096a50 0x7f726c132680 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.024+0000 7f7271ca6700 1 -- 172.21.15.7:0/153742073 >> 172.21.15.7:0/153742073 conn(0x7f726c070cf0 msgr2=0x7f726c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:59.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.024+0000 7f7271ca6700 1 -- 172.21.15.7:0/153742073 shutdown_connections 2023-09-30T01:59:59.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.024+0000 7f7271ca6700 1 -- 172.21.15.7:0/153742073 wait complete. 2023-09-30T01:59:59.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.024+0000 7f7271ca6700 10 monclient: build_initial_monmap 2023-09-30T01:59:59.033 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.024+0000 7f7271ca6700 10 monclient: monmap: 2023-09-30T01:59:59.033 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:59.033 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:59.033 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:59.033 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:59.033 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:59.033 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.033 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:59.034 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:59.034 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:59.034 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f7271ca6700 1 Processor -- start 2023-09-30T01:59:59.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f7271ca6700 1 -- start start 2023-09-30T01:59:59.034 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f7271ca6700 10 monclient: init 2023-09-30T01:59:59.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f7271ca6700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:59.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f7271ca6700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T01:59:59.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f7271ca6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f726c096a50 0x7f726c12fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f7271ca6700 10 monclient(hunting): picked mon.noname-b con 0x7f726c096a50 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:59.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f7271ca6700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f726c096e20 0x7f726c09e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.038 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f7271ca6700 10 monclient(hunting): picked mon.noname-c con 0x7f726c096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:59.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f7271ca6700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f726c137710 0x7f726c09ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f7271ca6700 10 monclient(hunting): picked mon.noname-a con 0x7f726c137710 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:59.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f7271ca6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:59.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f7271ca6700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726c0a0e30 con 0x7f726c137710 2023-09-30T01:59:59.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f7271ca6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:59.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f7271ca6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726c131d60 con 0x7f726c096a50 2023-09-30T01:59:59.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f7271ca6700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:59.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f7271ca6700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f726c1334e0 con 0x7f726c096e20 2023-09-30T01:59:59.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f7271ca6700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:59.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f7271ca6700 10 monclient(hunting): authenticate will time out at 1488.090884s 2023-09-30T01:59:59.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f726affd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f726c096e20 0x7f726c09e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.039 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f726bfff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f726c137710 0x7f726c09ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f726affd700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f726c096e20 0x7f726c09e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41464/0 (socket says 172.21.15.7:41464) 2023-09-30T01:59:59.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f726affd700 1 -- 172.21.15.7:0/1773973342 learned_addr learned my addr 172.21.15.7:0/1773973342 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:59.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f726affd700 10 monclient(hunting): get_auth_request con 0x7f726c096e20 auth_method 0 2023-09-30T01:59:59.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f726affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:59.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f726affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:59.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.025+0000 7f726affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:59.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726bfff700 10 monclient(hunting): get_auth_request con 0x7f726c137710 auth_method 0 2023-09-30T01:59:59.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:59.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:59.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:59.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726b7fe700 1 --2- 172.21.15.7:0/1773973342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f726c096a50 0x7f726c12fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.040 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:59.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:59.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:59.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726b7fe700 10 monclient(hunting): get_auth_request con 0x7f726c096a50 auth_method 0 2023-09-30T01:59:59.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:59.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:59.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:59.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:59.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:59.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:59.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:59.041 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:59.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:59.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726affd700 10 monclient(hunting): handle_auth_done global_id 4823 payload 824 2023-09-30T01:59:59.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726affd700 1 -- 172.21.15.7:0/1773973342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f726c096a50 msgr2=0x7f726c12fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:59.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726affd700 1 --2- 172.21.15.7:0/1773973342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f726c096a50 0x7f726c12fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726affd700 1 -- 172.21.15.7:0/1773973342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f726c137710 msgr2=0x7f726c09ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:59.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726affd700 1 --2- 172.21.15.7:0/1773973342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f726c137710 0x7f726c09ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726affd700 10 monclient: _finish_hunting 0 2023-09-30T01:59:59.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726affd700 1 monclient: found mon.noname-c 2023-09-30T01:59:59.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:59.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726affd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:59:59.042 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.026+0000 7f726affd700 1 -- 172.21.15.7:0/1773973342 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725c009420 con 0x7f726c096e20 2023-09-30T01:59:59.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f726affd700 1 -- 172.21.15.7:0/1773973342 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f726c096e20 msgr2=0x7f726c09e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:59.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f726affd700 1 --2- 172.21.15.7:0/1773973342 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f726c096e20 0x7f726c09e790 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f726000c5e0 tx=0x7f7260004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:59.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f72537fe700 1 -- 172.21.15.7:0/1773973342 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7260005d40 con 0x7f726c096e20 2023-09-30T01:59:59.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f72537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:59.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f72537fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:59:59.043 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f72537fe700 10 monclient: dump: 2023-09-30T01:59:59.043 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:59.043 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:59.043 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.043 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.043 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:59.043 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.044 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:59.044 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:59.044 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:59.044 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f72537fe700 10 monclient: _finish_auth 0 2023-09-30T01:59:59.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f72537fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:59.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f7271ca6700 5 monclient: authenticate success, global_id 4823 2023-09-30T01:59:59.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f7271ca6700 10 monclient: _renew_subs 2023-09-30T01:59:59.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f72537fe700 1 -- 172.21.15.7:0/1773973342 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72600070d0 con 0x7f726c096e20 2023-09-30T01:59:59.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f7271ca6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:59.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f7271ca6700 1 -- 172.21.15.7:0/1773973342 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f726c0a1000 con 0x7f726c096e20 2023-09-30T01:59:59.044 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f72537fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f72537fe700 1 -- 172.21.15.7:0/1773973342 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7260007490 con 0x7f726c096e20 2023-09-30T01:59:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f72537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f72537fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f72537fe700 10 monclient: dump: 2023-09-30T01:59:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:59.045 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:59.046 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:59.046 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f7269ffb700 4 set_mon_vals no callback set 2023-09-30T01:59:59.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f7271ca6700 10 monclient: _renew_subs 2023-09-30T01:59:59.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f7271ca6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:59.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.028+0000 7f7271ca6700 1 -- 172.21.15.7:0/1773973342 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f726c137f80 con 0x7f726c096e20 2023-09-30T01:59:59.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.029+0000 7f72537fe700 1 -- 172.21.15.7:0/1773973342 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f726001b3b0 con 0x7f726c096e20 2023-09-30T01:59:59.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.029+0000 7f7271ca6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:59.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.029+0000 7f7271ca6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:59.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.029+0000 7f7271ca6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:59.046 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.029+0000 7f7271ca6700 1 -- 172.21.15.7:0/1773973342 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f726c11c200 con 0x7f726c096e20 2023-09-30T01:59:59.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.029+0000 7f72537fe700 1 --2- 172.21.15.7:0/1773973342 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7258035810 0x7f7258037cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.029+0000 7f72537fe700 1 -- 172.21.15.7:0/1773973342 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f7260047110 con 0x7f726c096e20 2023-09-30T01:59:59.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.029+0000 7f726b7fe700 1 --2- 172.21.15.7:0/1773973342 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7258035810 0x7f7258037cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.029+0000 7f726b7fe700 10 monclient: get_auth_request con 0x7f7258035810 auth_method 0 2023-09-30T01:59:59.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.032+0000 7f726b7fe700 1 --2- 172.21.15.7:0/1773973342 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7258035810 0x7f7258037cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f725c0030c0 tx=0x7f725c016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:59.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.034+0000 7f72537fe700 1 -- 172.21.15.7:0/1773973342 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f726c11c200 con 0x7f726c096e20 2023-09-30T01:59:59.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.034+0000 7f72537fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:59.047 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.034+0000 7f72537fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:59.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.228+0000 7f7271ca6700 1 -- 172.21.15.7:0/1773973342 --> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f726c11c200 con 0x7f7258035810 2023-09-30T01:59:59.229 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.229+0000 7f72537fe700 1 -- 172.21.15.7:0/1773973342 <== mgr.4105 v2:172.21.15.50:6824/86200 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f726c11c200 con 0x7f7258035810 2023-09-30T01:59:59.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.231+0000 7f7271ca6700 1 -- 172.21.15.7:0/1773973342 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7258035810 msgr2=0x7f7258037cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:59.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.231+0000 7f7271ca6700 1 --2- 172.21.15.7:0/1773973342 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7258035810 0x7f7258037cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f725c0030c0 tx=0x7f725c016000 comp rx=0 tx=0).stop 2023-09-30T01:59:59.231 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.231+0000 7f7271ca6700 10 monclient: shutdown 2023-09-30T01:59:59.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.231+0000 7f7271ca6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:59.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.231+0000 7f7271ca6700 1 -- 172.21.15.7:0/1773973342 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f726c096e20 msgr2=0x7f726c09e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:59.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.231+0000 7f7271ca6700 1 --2- 172.21.15.7:0/1773973342 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f726c096e20 0x7f726c09e790 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f726000c5e0 tx=0x7f7260004a80 comp rx=0 tx=0).stop 2023-09-30T01:59:59.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.231+0000 7f7271ca6700 1 -- 172.21.15.7:0/1773973342 shutdown_connections 2023-09-30T01:59:59.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.231+0000 7f7271ca6700 1 --2- 172.21.15.7:0/1773973342 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f7258035810 0x7f7258037cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.231+0000 7f7271ca6700 1 --2- 172.21.15.7:0/1773973342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f726c137710 0x7f726c09ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.231+0000 7f7271ca6700 1 --2- 172.21.15.7:0/1773973342 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f726c096a50 0x7f726c12fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.231+0000 7f7271ca6700 1 --2- 172.21.15.7:0/1773973342 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f726c096e20 0x7f726c09e790 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.231+0000 7f7271ca6700 1 -- 172.21.15.7:0/1773973342 >> 172.21.15.7:0/1773973342 conn(0x7f726c070cf0 msgr2=0x7f726c1336f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:59.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.231+0000 7f7271ca6700 1 -- 172.21.15.7:0/1773973342 shutdown_connections 2023-09-30T01:59:59.232 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.231+0000 7f7271ca6700 1 -- 172.21.15.7:0/1773973342 wait complete. 2023-09-30T01:59:59.233 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 4 to deep-scrub 2023-09-30T01:59:59.243 INFO:tasks.ceph:Scrubbing osd.5 2023-09-30T01:59:59.243 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-09-30T01:59:59.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.364+0000 7fb63e849700 10 monclient: get_monmap_and_config 2023-09-30T01:59:59.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.364+0000 7fb63e849700 10 monclient: build_initial_monmap 2023-09-30T01:59:59.364 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.364+0000 7fb63e849700 10 monclient: monmap: 2023-09-30T01:59:59.365 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:59.365 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:59.365 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:59.365645+0000 2023-09-30T01:59:59.365 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:59.365645+0000 2023-09-30T01:59:59.365 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:59.365 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.365 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:59.365 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:59.365 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:59.365 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:59.365 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:59.366 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.365+0000 7fb63e849700 1 Processor -- start 2023-09-30T01:59:59.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.365+0000 7fb63e849700 1 -- start start 2023-09-30T01:59:59.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.365+0000 7fb63e849700 10 monclient: init 2023-09-30T01:59:59.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.365+0000 7fb63e849700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:59.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.365+0000 7fb63e849700 10 monclient: _add_conns ranks=[4,3,2,0,1] 2023-09-30T01:59:59.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.365+0000 7fb63e849700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7fb6380985c0 0x7fb638132690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.365+0000 7fb63e849700 10 monclient(hunting): picked mon.noname-c con 0x7fb6380985c0 addr v2:172.21.15.50:3300/0 2023-09-30T01:59:59.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.365+0000 7fb63e849700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb638098990 0x7fb638098da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.370 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.365+0000 7fb63e849700 10 monclient(hunting): picked mon.noname-a con 0x7fb638098990 addr v2:172.21.15.7:3300/0 2023-09-30T01:59:59.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.365+0000 7fb63e849700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb63806a030 0x7fb63806a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.365+0000 7fb63e849700 10 monclient(hunting): picked mon.noname-b con 0x7fb63806a030 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:59.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.365+0000 7fb63e849700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:59.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.365+0000 7fb63e849700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6380a0800 con 0x7fb638098990 2023-09-30T01:59:59.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.365+0000 7fb63e849700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:59.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.365+0000 7fb63e849700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb638096b80 con 0x7fb6380985c0 2023-09-30T01:59:59.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.365+0000 7fb63e849700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:59.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.365+0000 7fb63e849700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6380a1c90 con 0x7fb63806a030 2023-09-30T01:59:59.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.365+0000 7fb63e849700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:59.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.365+0000 7fb63e849700 10 monclient(hunting): authenticate will time out at 1488.430833s 2023-09-30T01:59:59.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.365+0000 7fb63e048700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb63806a030 0x7fb63806a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.365+0000 7fb63d046700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb638098990 0x7fb638098da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63e048700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb63806a030 0x7fb63806a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41468/0 (socket says 172.21.15.7:41468) 2023-09-30T01:59:59.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63d046700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7fb638098990 0x7fb638098da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49464/0 (socket says 172.21.15.7:49464) 2023-09-30T01:59:59.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63e048700 1 -- 172.21.15.7:0/1827819617 learned_addr learned my addr 172.21.15.7:0/1827819617 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:59.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63e048700 10 monclient(hunting): get_auth_request con 0x7fb63806a030 auth_method 0 2023-09-30T01:59:59.372 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63e048700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:59.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63e048700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:59.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63e048700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:59.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63d046700 10 monclient(hunting): get_auth_request con 0x7fb638098990 auth_method 0 2023-09-30T01:59:59.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63d046700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:59.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63d046700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:59.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63d046700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:59.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63e048700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:59.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63e048700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:59.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63d847700 1 --2- 172.21.15.7:0/1827819617 >> v2:172.21.15.50:3300/0 conn(0x7fb6380985c0 0x7fb638132690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63e048700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:59.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63d046700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63d046700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63d046700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63d847700 10 monclient(hunting): get_auth_request con 0x7fb6380985c0 auth_method 0 2023-09-30T01:59:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63d847700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63d847700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63d847700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63d847700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63d847700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63d847700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63e048700 10 monclient(hunting): handle_auth_done global_id 4826 payload 274 2023-09-30T01:59:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63e048700 1 -- 172.21.15.7:0/1827819617 >> v2:172.21.15.50:3300/0 conn(0x7fb6380985c0 msgr2=0x7fb638132690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63e048700 1 --2- 172.21.15.7:0/1827819617 >> v2:172.21.15.50:3300/0 conn(0x7fb6380985c0 0x7fb638132690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63e048700 1 -- 172.21.15.7:0/1827819617 >> v2:172.21.15.7:3300/0 conn(0x7fb638098990 msgr2=0x7fb638098da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63e048700 1 --2- 172.21.15.7:0/1827819617 >> v2:172.21.15.7:3300/0 conn(0x7fb638098990 0x7fb638098da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63e048700 10 monclient: _finish_hunting 0 2023-09-30T01:59:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63e048700 1 monclient: found mon.noname-b 2023-09-30T01:59:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63e048700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63e048700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.366+0000 7fb63e048700 1 -- 172.21.15.7:0/1827819617 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6380a19b0 con 0x7fb63806a030 2023-09-30T01:59:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.368+0000 7fb63e048700 1 -- 172.21.15.7:0/1827819617 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb63806a030 msgr2=0x7fb63806a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.368+0000 7fb63e048700 1 --2- 172.21.15.7:0/1827819617 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb63806a030 0x7fb63806a490 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb63806b750 tx=0x7fb630009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2833cc1c2099a61 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.368+0000 7fb63c845700 1 -- 172.21.15.7:0/1827819617 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63000b850 con 0x7fb63806a030 2023-09-30T01:59:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.368+0000 7fb63c845700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:59.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.368+0000 7fb63c845700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:59.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.368+0000 7fb63c845700 10 monclient: dump: 2023-09-30T01:59:59.376 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:59.376 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:59.376 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.376 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.376 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:59.376 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.376 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:59.376 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:59.376 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:59.376 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.376 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.368+0000 7fb63c845700 10 monclient: _finish_auth 0 2023-09-30T01:59:59.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.368+0000 7fb63c845700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:59.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.368+0000 7fb63c845700 1 -- 172.21.15.7:0/1827819617 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63000b9b0 con 0x7fb63806a030 2023-09-30T01:59:59.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.368+0000 7fb63e849700 5 monclient: authenticate success, global_id 4826 2023-09-30T01:59:59.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.368+0000 7fb63e849700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:59.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.368+0000 7fb63c845700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:59.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.368+0000 7fb63e849700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:59.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.368+0000 7fb63c845700 1 -- 172.21.15.7:0/1827819617 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63000bd50 con 0x7fb63806a030 2023-09-30T01:59:59.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.368+0000 7fb63c845700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:59.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.368+0000 7fb63c845700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:59.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.368+0000 7fb63e849700 4 set_mon_vals no callback set 2023-09-30T01:59:59.377 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.368+0000 7fb63c845700 10 monclient: dump: 2023-09-30T01:59:59.377 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:59.378 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:59.378 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.378 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.378 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:59.378 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.378 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:59.378 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:59.378 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:59.378 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.369+0000 7fb63e849700 10 monclient: shutdown 2023-09-30T01:59:59.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.369+0000 7fb63e849700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:59.378 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.369+0000 7fb63e849700 1 -- 172.21.15.7:0/1827819617 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb63806a030 msgr2=0x7fb63806a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:59.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.369+0000 7fb63e849700 1 --2- 172.21.15.7:0/1827819617 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb63806a030 0x7fb63806a490 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb63806b750 tx=0x7fb630009250 comp rx=0 tx=0).stop 2023-09-30T01:59:59.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.369+0000 7fb63e849700 1 -- 172.21.15.7:0/1827819617 shutdown_connections 2023-09-30T01:59:59.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.369+0000 7fb63e849700 1 --2- 172.21.15.7:0/1827819617 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb63806a030 0x7fb63806a490 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.369+0000 7fb63e849700 1 --2- 172.21.15.7:0/1827819617 >> v2:172.21.15.50:3300/0 conn(0x7fb6380985c0 0x7fb638132690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.369+0000 7fb63e849700 1 --2- 172.21.15.7:0/1827819617 >> v2:172.21.15.7:3300/0 conn(0x7fb638098990 0x7fb638098da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.369+0000 7fb63e849700 1 -- 172.21.15.7:0/1827819617 >> 172.21.15.7:0/1827819617 conn(0x7fb6380a2070 msgr2=0x7fb6380a2470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:59.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.369+0000 7fb63e849700 1 -- 172.21.15.7:0/1827819617 shutdown_connections 2023-09-30T01:59:59.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.369+0000 7fb63e849700 1 -- 172.21.15.7:0/1827819617 wait complete. 2023-09-30T01:59:59.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.370+0000 7fb63e849700 10 monclient: build_initial_monmap 2023-09-30T01:59:59.379 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.370+0000 7fb63e849700 10 monclient: monmap: 2023-09-30T01:59:59.379 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:59.380 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:59.380 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:59.380 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:59.380 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:59.380 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.380 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:59.380 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:59.380 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:59.380 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.370+0000 7fb63e849700 1 Processor -- start 2023-09-30T01:59:59.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.370+0000 7fb63e849700 1 -- start start 2023-09-30T01:59:59.380 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.370+0000 7fb63e849700 10 monclient: init 2023-09-30T01:59:59.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.370+0000 7fb63e849700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:59.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.370+0000 7fb63e849700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:59:59.385 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.370+0000 7fb63e849700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb63806a030 0x7fb6381411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.370+0000 7fb63e849700 10 monclient(hunting): picked mon.noname-c con 0x7fb63806a030 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:59.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.370+0000 7fb63e849700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6380985c0 0x7fb63812f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.370+0000 7fb63e849700 10 monclient(hunting): picked mon.noname-b con 0x7fb6380985c0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:59.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.370+0000 7fb63e849700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb638098990 0x7fb63812f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.370+0000 7fb63e849700 10 monclient(hunting): picked mon.noname-a con 0x7fb638098990 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:59.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.370+0000 7fb63e849700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:59.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.370+0000 7fb63e849700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6380a12b0 con 0x7fb638098990 2023-09-30T01:59:59.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.370+0000 7fb63e849700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:59.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.370+0000 7fb63e849700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63811fd00 con 0x7fb6380985c0 2023-09-30T01:59:59.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.370+0000 7fb63e849700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:59.386 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.370+0000 7fb63e849700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb63811f460 con 0x7fb63806a030 2023-09-30T01:59:59.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.370+0000 7fb63e849700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:59.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.370+0000 7fb63e849700 10 monclient(hunting): authenticate will time out at 1488.435990s 2023-09-30T01:59:59.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63d847700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb63806a030 0x7fb6381411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63e048700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb638098990 0x7fb63812f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63d847700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb63806a030 0x7fb6381411c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41482/0 (socket says 172.21.15.7:41482) 2023-09-30T01:59:59.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63e048700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb638098990 0x7fb63812f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49466/0 (socket says 172.21.15.7:49466) 2023-09-30T01:59:59.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63d847700 1 -- 172.21.15.7:0/2659781995 learned_addr learned my addr 172.21.15.7:0/2659781995 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:59.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63d847700 10 monclient(hunting): get_auth_request con 0x7fb63806a030 auth_method 0 2023-09-30T01:59:59.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63d847700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:59.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63d847700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:59.387 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63d847700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:59.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63e048700 10 monclient(hunting): get_auth_request con 0x7fb638098990 auth_method 0 2023-09-30T01:59:59.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63e048700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:59.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63e048700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:59.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63e048700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:59.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63e048700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:59.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63e048700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:59.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63e048700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:59.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63d847700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:59.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63d847700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:59.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63d847700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:59.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63e048700 10 monclient(hunting): handle_auth_done global_id 4806 payload 824 2023-09-30T01:59:59.388 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63d046700 1 --2- 172.21.15.7:0/2659781995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6380985c0 0x7fb63812f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63e048700 1 -- 172.21.15.7:0/2659781995 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb63806a030 msgr2=0x7fb6381411c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:59.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63e048700 1 --2- 172.21.15.7:0/2659781995 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7fb63806a030 0x7fb6381411c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63e048700 1 -- 172.21.15.7:0/2659781995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6380985c0 msgr2=0x7fb63812f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:59.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63e048700 1 --2- 172.21.15.7:0/2659781995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6380985c0 0x7fb63812f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63e048700 10 monclient: _finish_hunting 0 2023-09-30T01:59:59.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63e048700 1 monclient: found mon.noname-a 2023-09-30T01:59:59.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63e048700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:59.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63e048700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T01:59:59.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.371+0000 7fb63e048700 1 -- 172.21.15.7:0/2659781995 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6300094f0 con 0x7fb638098990 2023-09-30T01:59:59.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.373+0000 7fb63e048700 1 -- 172.21.15.7:0/2659781995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb638098990 msgr2=0x7fb63812f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:59.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.373+0000 7fb63e048700 1 --2- 172.21.15.7:0/2659781995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb638098990 0x7fb63812f850 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb63806b930 tx=0x7fb630017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:59.389 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.373+0000 7fb635ffb700 1 -- 172.21.15.7:0/2659781995 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb630011e20 con 0x7fb638098990 2023-09-30T01:59:59.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.373+0000 7fb635ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:59.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.373+0000 7fb635ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T01:59:59.390 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.373+0000 7fb635ffb700 10 monclient: dump: 2023-09-30T01:59:59.390 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:59.390 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:59.390 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.390 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.390 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:59.390 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.390 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:59.390 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:59.390 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:59.391 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.373+0000 7fb635ffb700 10 monclient: _finish_auth 0 2023-09-30T01:59:59.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.373+0000 7fb635ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:59.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.373+0000 7fb635ffb700 1 -- 172.21.15.7:0/2659781995 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb630006a90 con 0x7fb638098990 2023-09-30T01:59:59.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.373+0000 7fb63e849700 5 monclient: authenticate success, global_id 4806 2023-09-30T01:59:59.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.373+0000 7fb63e849700 10 monclient: _renew_subs 2023-09-30T01:59:59.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.373+0000 7fb63e849700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:59.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.373+0000 7fb63e849700 1 -- 172.21.15.7:0/2659781995 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb63811f740 con 0x7fb638098990 2023-09-30T01:59:59.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.373+0000 7fb635ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:59.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.373+0000 7fb635ffb700 1 -- 172.21.15.7:0/2659781995 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63001a880 con 0x7fb638098990 2023-09-30T01:59:59.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.373+0000 7fb635ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:59.391 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.373+0000 7fb635ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T01:59:59.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.373+0000 7fb635ffb700 10 monclient: dump: 2023-09-30T01:59:59.392 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:59.392 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:59.392 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.392 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.392 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:59.392 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.392 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:59.392 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:59.392 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:59.392 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.392 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.374+0000 7fb63e849700 10 monclient: _renew_subs 2023-09-30T01:59:59.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.374+0000 7fb63e849700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:59.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.374+0000 7fb63e849700 1 -- 172.21.15.7:0/2659781995 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb638141700 con 0x7fb638098990 2023-09-30T01:59:59.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.373+0000 7fb637fff700 4 set_mon_vals no callback set 2023-09-30T01:59:59.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.374+0000 7fb63e849700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T01:59:59.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.374+0000 7fb63e849700 1 -- 172.21.15.7:0/2659781995 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb63805f230 con 0x7fb638098990 2023-09-30T01:59:59.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.374+0000 7fb635ffb700 1 -- 172.21.15.7:0/2659781995 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7fb63001ac30 con 0x7fb638098990 2023-09-30T01:59:59.393 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.374+0000 7fb635ffb700 1 --2- 172.21.15.7:0/2659781995 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb6240357e0 0x7fb624037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.375+0000 7fb635ffb700 1 -- 172.21.15.7:0/2659781995 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7fb6300496c0 con 0x7fb638098990 2023-09-30T01:59:59.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.375+0000 7fb635ffb700 1 --2- 172.21.15.7:0/2659781995 >> [v2:172.21.15.50:6800/86634,v1:172.21.15.50:6801/86634] conn(0x7fb62403b660 0x7fb62403da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.375+0000 7fb635ffb700 1 -- 172.21.15.7:0/2659781995 --> [v2:172.21.15.50:6800/86634,v1:172.21.15.50:6801/86634] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb62403e120 con 0x7fb62403b660 2023-09-30T01:59:59.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.375+0000 7fb635ffb700 1 -- 172.21.15.7:0/2659781995 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb63004bc50 con 0x7fb638098990 2023-09-30T01:59:59.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.375+0000 7fb635ffb700 10 monclient: handle_get_version_reply finishing 1 version 15 2023-09-30T01:59:59.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.375+0000 7fb63d847700 1 --2- 172.21.15.7:0/2659781995 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb6240357e0 0x7fb624037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.375+0000 7fb63d847700 10 monclient: get_auth_request con 0x7fb6240357e0 auth_method 0 2023-09-30T01:59:59.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.375+0000 7fb63d046700 1 --2- 172.21.15.7:0/2659781995 >> [v2:172.21.15.50:6800/86634,v1:172.21.15.50:6801/86634] conn(0x7fb62403b660 0x7fb62403da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.375+0000 7fb63d046700 10 monclient: get_auth_request con 0x7fb62403b660 auth_method 0 2023-09-30T01:59:59.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.377+0000 7fb63d847700 1 --2- 172.21.15.7:0/2659781995 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb6240357e0 0x7fb624037c90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb6280099f0 tx=0x7fb62800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:59.394 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.380+0000 7fb63d046700 1 --2- 172.21.15.7:0/2659781995 >> [v2:172.21.15.50:6800/86634,v1:172.21.15.50:6801/86634] conn(0x7fb62403b660 0x7fb62403da70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:59.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.380+0000 7fb635ffb700 1 -- 172.21.15.7:0/2659781995 <== osd.5 v2:172.21.15.50:6800/86634 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fb62403e120 con 0x7fb62403b660 2023-09-30T01:59:59.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.394+0000 7fb63e849700 1 -- 172.21.15.7:0/2659781995 --> [v2:172.21.15.50:6800/86634,v1:172.21.15.50:6801/86634] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb6380695f0 con 0x7fb62403b660 2023-09-30T01:59:59.395 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.395+0000 7fb635ffb700 1 -- 172.21.15.7:0/2659781995 <== osd.5 v2:172.21.15.50:6800/86634 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fb6380695f0 con 0x7fb62403b660 2023-09-30T01:59:59.396 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-09-30T01:59:59.396 INFO:teuthology.orchestra.run.smithi007.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T01:59:59.396 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-09-30T01:59:59.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.395+0000 7fb63e849700 1 -- 172.21.15.7:0/2659781995 >> [v2:172.21.15.50:6800/86634,v1:172.21.15.50:6801/86634] conn(0x7fb62403b660 msgr2=0x7fb62403da70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:59.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.395+0000 7fb63e849700 1 --2- 172.21.15.7:0/2659781995 >> [v2:172.21.15.50:6800/86634,v1:172.21.15.50:6801/86634] conn(0x7fb62403b660 0x7fb62403da70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.395+0000 7fb63e849700 1 -- 172.21.15.7:0/2659781995 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb6240357e0 msgr2=0x7fb624037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:59.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.395+0000 7fb63e849700 1 --2- 172.21.15.7:0/2659781995 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7fb6240357e0 0x7fb624037c90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb6280099f0 tx=0x7fb62800b040 comp rx=0 tx=0).stop 2023-09-30T01:59:59.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.395+0000 7fb63e849700 10 monclient: shutdown 2023-09-30T01:59:59.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.395+0000 7fb63e849700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:59.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.395+0000 7fb63e849700 1 -- 172.21.15.7:0/2659781995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb638098990 msgr2=0x7fb63812f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:59.397 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.395+0000 7fb63e849700 1 --2- 172.21.15.7:0/2659781995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb638098990 0x7fb63812f850 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb63806b930 tx=0x7fb630017000 comp rx=0 tx=0).stop 2023-09-30T01:59:59.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.395+0000 7fb63e048700 1 -- 172.21.15.7:0/2659781995 reap_dead start 2023-09-30T01:59:59.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.395+0000 7fb63e849700 1 -- 172.21.15.7:0/2659781995 shutdown_connections 2023-09-30T01:59:59.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.395+0000 7fb63e849700 1 -- 172.21.15.7:0/2659781995 >> 172.21.15.7:0/2659781995 conn(0x7fb6380a2070 msgr2=0x7fb638097310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:59.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.395+0000 7fb63e849700 1 -- 172.21.15.7:0/2659781995 shutdown_connections 2023-09-30T01:59:59.398 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.395+0000 7fb63e849700 1 -- 172.21.15.7:0/2659781995 wait complete. 2023-09-30T01:59:59.407 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-09-30T01:59:59.525 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.525+0000 7f8f637fe700 10 monclient: get_monmap_and_config 2023-09-30T01:59:59.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.525+0000 7f8f637fe700 10 monclient: build_initial_monmap 2023-09-30T01:59:59.526 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.525+0000 7f8f637fe700 10 monclient: monmap: 2023-09-30T01:59:59.526 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:59.526 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:59.526 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:59.526962+0000 2023-09-30T01:59:59.527 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:59.526962+0000 2023-09-30T01:59:59.527 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:59.527 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.527 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:59.527 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:59.527 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:59.527 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:59.527 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:59.527 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.526+0000 7f8f637fe700 1 Processor -- start 2023-09-30T01:59:59.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.526+0000 7f8f637fe700 1 -- start start 2023-09-30T01:59:59.537 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.526+0000 7f8f637fe700 10 monclient: init 2023-09-30T01:59:59.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f637fe700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:59.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f637fe700 10 monclient: _add_conns ranks=[3,2,4,1,0] 2023-09-30T01:59:59.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f637fe700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8f5c096a50 0x7f8f5c132680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f637fe700 10 monclient(hunting): picked mon.noname-a con 0x7f8f5c096a50 addr v2:172.21.15.7:3300/0 2023-09-30T01:59:59.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f637fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f5c096e20 0x7f8f5c0971f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f637fe700 10 monclient(hunting): picked mon.noname-b con 0x7f8f5c096e20 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:59.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f637fe700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f8f5c12ef00 0x7f8f5c12f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f637fe700 10 monclient(hunting): picked mon.noname-c con 0x7f8f5c12ef00 addr v2:172.21.15.50:3300/0 2023-09-30T01:59:59.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f637fe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:59.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f637fe700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f5c099440 con 0x7f8f5c096a50 2023-09-30T01:59:59.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f637fe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:59.538 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f637fe700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f5c097b40 con 0x7f8f5c12ef00 2023-09-30T01:59:59.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f637fe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:59.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f637fe700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c099160 con 0x7f8f5c096e20 2023-09-30T01:59:59.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f637fe700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:59.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f637fe700 10 monclient(hunting): authenticate will time out at 1488.592181s 2023-09-30T01:59:59.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f6159a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8f5c096a50 0x7f8f5c132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f60d99700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f5c096e20 0x7f8f5c0971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f6159a700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f8f5c096a50 0x7f8f5c132680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49472/0 (socket says 172.21.15.7:49472) 2023-09-30T01:59:59.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f60d99700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f5c096e20 0x7f8f5c0971f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41488/0 (socket says 172.21.15.7:41488) 2023-09-30T01:59:59.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f6159a700 1 -- 172.21.15.7:0/1897678910 learned_addr learned my addr 172.21.15.7:0/1897678910 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:59.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f6159a700 10 monclient(hunting): get_auth_request con 0x7f8f5c096a50 auth_method 0 2023-09-30T01:59:59.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f6159a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:59.539 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f6159a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:59.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f6159a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:59.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f60d99700 10 monclient(hunting): get_auth_request con 0x7f8f5c096e20 auth_method 0 2023-09-30T01:59:59.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f60d99700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:59.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f60d99700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:59.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f60d99700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:59.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f60d99700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:59.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f60d99700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:59.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f60d99700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:59.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f6159a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:59.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f6159a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:59.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f6159a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:59.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f61d9b700 1 --2- 172.21.15.7:0/1897678910 >> v2:172.21.15.50:3300/0 conn(0x7f8f5c12ef00 0x7f8f5c12f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f60d99700 10 monclient(hunting): handle_auth_done global_id 4832 payload 274 2023-09-30T01:59:59.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f60d99700 1 -- 172.21.15.7:0/1897678910 >> v2:172.21.15.50:3300/0 conn(0x7f8f5c12ef00 msgr2=0x7f8f5c12f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:59.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f60d99700 1 --2- 172.21.15.7:0/1897678910 >> v2:172.21.15.50:3300/0 conn(0x7f8f5c12ef00 0x7f8f5c12f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f60d99700 1 -- 172.21.15.7:0/1897678910 >> v2:172.21.15.7:3300/0 conn(0x7f8f5c096a50 msgr2=0x7f8f5c132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:59.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f60d99700 1 --2- 172.21.15.7:0/1897678910 >> v2:172.21.15.7:3300/0 conn(0x7f8f5c096a50 0x7f8f5c132680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f60d99700 10 monclient: _finish_hunting 0 2023-09-30T01:59:59.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f60d99700 1 monclient: found mon.noname-b 2023-09-30T01:59:59.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f60d99700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:59.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f60d99700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:59.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f60d99700 1 -- 172.21.15.7:0/1897678910 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5c099720 con 0x7f8f5c096e20 2023-09-30T01:59:59.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f61d9b700 10 monclient: get_auth_request con 0x7f8f5c12ef00 auth_method 0 2023-09-30T01:59:59.541 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f61d9b700 1 --2- 172.21.15.7:0/1897678910 >> v2:172.21.15.50:3300/0 conn(0x7f8f5c12ef00 0x7f8f5c12f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:59:59.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.527+0000 7f8f6159a700 1 --2- 172.21.15.7:0/1897678910 >> v2:172.21.15.7:3300/0 conn(0x7f8f5c096a50 0x7f8f5c132680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:59.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.529+0000 7f8f60d99700 1 -- 172.21.15.7:0/1897678910 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f5c096e20 msgr2=0x7f8f5c0971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:59.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.529+0000 7f8f60d99700 1 --2- 172.21.15.7:0/1897678910 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f5c096e20 0x7f8f5c0971f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c0a5710 tx=0x7f8f500091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e31e6d0438bd795 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:59.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f5bfff700 1 -- 172.21.15.7:0/1897678910 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f50011840 con 0x7f8f5c096e20 2023-09-30T01:59:59.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:59.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f5bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:59.542 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f5bfff700 10 monclient: dump: 2023-09-30T01:59:59.542 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:59.542 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:59.542 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.542 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.543 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:59.543 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.543 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:59.543 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:59.543 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:59.543 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f5bfff700 10 monclient: _finish_auth 0 2023-09-30T01:59:59.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f5bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:59.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f637fe700 5 monclient: authenticate success, global_id 4832 2023-09-30T01:59:59.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f5bfff700 1 -- 172.21.15.7:0/1897678910 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f500119a0 con 0x7f8f5c096e20 2023-09-30T01:59:59.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f637fe700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:59.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f5bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:59.543 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f637fe700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:59.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f637fe700 4 set_mon_vals no callback set 2023-09-30T01:59:59.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f5bfff700 1 -- 172.21.15.7:0/1897678910 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f50011d40 con 0x7f8f5c096e20 2023-09-30T01:59:59.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:59.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f5bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:59.544 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f5bfff700 10 monclient: dump: 2023-09-30T01:59:59.544 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:59.544 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:59.544 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.544 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.544 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:59.544 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.544 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:59.545 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:59.545 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:59.545 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f637fe700 10 monclient: shutdown 2023-09-30T01:59:59.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f637fe700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:59.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f637fe700 1 -- 172.21.15.7:0/1897678910 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f5c096e20 msgr2=0x7f8f5c0971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:59.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f637fe700 1 --2- 172.21.15.7:0/1897678910 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f5c096e20 0x7f8f5c0971f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c0a5710 tx=0x7f8f500091a0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f637fe700 1 -- 172.21.15.7:0/1897678910 shutdown_connections 2023-09-30T01:59:59.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f637fe700 1 --2- 172.21.15.7:0/1897678910 >> v2:172.21.15.50:3300/0 conn(0x7f8f5c12ef00 0x7f8f5c12f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f637fe700 1 --2- 172.21.15.7:0/1897678910 >> v2:172.21.15.7:3300/0 conn(0x7f8f5c096a50 0x7f8f5c132680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f637fe700 1 --2- 172.21.15.7:0/1897678910 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f5c096e20 0x7f8f5c0971f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.545 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f637fe700 1 -- 172.21.15.7:0/1897678910 >> 172.21.15.7:0/1897678910 conn(0x7f8f5c070cf0 msgr2=0x7f8f5c0710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:59.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f637fe700 1 -- 172.21.15.7:0/1897678910 shutdown_connections 2023-09-30T01:59:59.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.530+0000 7f8f637fe700 1 -- 172.21.15.7:0/1897678910 wait complete. 2023-09-30T01:59:59.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.531+0000 7f8f637fe700 10 monclient: build_initial_monmap 2023-09-30T01:59:59.546 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.531+0000 7f8f637fe700 10 monclient: monmap: 2023-09-30T01:59:59.546 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:59.546 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:59.546 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:59.546 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:59.546 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:59.547 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.547 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:59.547 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:59.547 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:59.547 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.531+0000 7f8f637fe700 1 Processor -- start 2023-09-30T01:59:59.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.531+0000 7f8f637fe700 1 -- start start 2023-09-30T01:59:59.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.531+0000 7f8f637fe700 10 monclient: init 2023-09-30T01:59:59.547 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f637fe700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:59.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f637fe700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:59:59.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f637fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f5c096a50 0x7f8f5c12d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f637fe700 10 monclient(hunting): picked mon.noname-c con 0x7f8f5c096a50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:59.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f637fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f5c096e20 0x7f8f5c12dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f637fe700 10 monclient(hunting): picked mon.noname-b con 0x7f8f5c096e20 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:59.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f637fe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f5c12ef00 0x7f8f5c09d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f637fe700 10 monclient(hunting): picked mon.noname-a con 0x7f8f5c12ef00 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:59.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f637fe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:59.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f637fe700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c1334e0 con 0x7f8f5c12ef00 2023-09-30T01:59:59.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f637fe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:59.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f637fe700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c0a4920 con 0x7f8f5c096e20 2023-09-30T01:59:59.548 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f637fe700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:59.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f637fe700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c09fb00 con 0x7f8f5c096a50 2023-09-30T01:59:59.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f637fe700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:59.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f637fe700 10 monclient(hunting): authenticate will time out at 1488.597233s 2023-09-30T01:59:59.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f6159a700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f5c096a50 0x7f8f5c12d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f6159a700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f5c096a50 0x7f8f5c12d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41502/0 (socket says 172.21.15.7:41502) 2023-09-30T01:59:59.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f61d9b700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f5c12ef00 0x7f8f5c09d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f6159a700 1 -- 172.21.15.7:0/4131827708 learned_addr learned my addr 172.21.15.7:0/4131827708 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:59.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f6159a700 10 monclient(hunting): get_auth_request con 0x7f8f5c096a50 auth_method 0 2023-09-30T01:59:59.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f6159a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:59.549 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f6159a700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:59.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f6159a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:59.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f61d9b700 10 monclient(hunting): get_auth_request con 0x7f8f5c12ef00 auth_method 0 2023-09-30T01:59:59.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f61d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:59.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f61d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:59.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f61d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:59.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f6159a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:59.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f6159a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:59.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f6159a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:59.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f61d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:59.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f61d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:59.550 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f61d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:59.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f60d99700 1 --2- 172.21.15.7:0/4131827708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f5c096e20 0x7f8f5c12dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f6159a700 10 monclient(hunting): handle_auth_done global_id 4835 payload 824 2023-09-30T01:59:59.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f6159a700 1 -- 172.21.15.7:0/4131827708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f5c096e20 msgr2=0x7f8f5c12dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:59.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f6159a700 1 --2- 172.21.15.7:0/4131827708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f5c096e20 0x7f8f5c12dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f6159a700 1 -- 172.21.15.7:0/4131827708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f5c12ef00 msgr2=0x7f8f5c09d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:59.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f6159a700 1 --2- 172.21.15.7:0/4131827708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f5c12ef00 0x7f8f5c09d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f6159a700 10 monclient: _finish_hunting 0 2023-09-30T01:59:59.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f6159a700 1 monclient: found mon.noname-c 2023-09-30T01:59:59.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f6159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:59.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f6159a700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:59:59.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f6159a700 1 -- 172.21.15.7:0/4131827708 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f540094f0 con 0x7f8f5c096a50 2023-09-30T01:59:59.551 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f60d99700 10 monclient: get_auth_request con 0x7f8f5c096e20 auth_method 0 2023-09-30T01:59:59.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.532+0000 7f8f60d99700 1 --2- 172.21.15.7:0/4131827708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f5c096e20 0x7f8f5c12dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T01:59:59.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.534+0000 7f8f6159a700 1 -- 172.21.15.7:0/4131827708 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f5c096a50 msgr2=0x7f8f5c12d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:59.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.534+0000 7f8f6159a700 1 --2- 172.21.15.7:0/4131827708 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f5c096a50 0x7f8f5c12d790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8f5400ec50 tx=0x7f8f54009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:59.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.534+0000 7f8f597fa700 1 -- 172.21.15.7:0/4131827708 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f540097d0 con 0x7f8f5c096a50 2023-09-30T01:59:59.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f597fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:59.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f597fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:59:59.552 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f597fa700 10 monclient: dump: 2023-09-30T01:59:59.552 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:59.552 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:59.552 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.552 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.552 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:59.553 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.553 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:59.553 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:59.553 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:59.553 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f597fa700 10 monclient: _finish_auth 0 2023-09-30T01:59:59.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f597fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:59.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f597fa700 1 -- 172.21.15.7:0/4131827708 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5400cb30 con 0x7f8f5c096a50 2023-09-30T01:59:59.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f637fe700 5 monclient: authenticate success, global_id 4835 2023-09-30T01:59:59.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f637fe700 10 monclient: _renew_subs 2023-09-30T01:59:59.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f637fe700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:59.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f637fe700 1 -- 172.21.15.7:0/4131827708 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f50009420 con 0x7f8f5c096a50 2023-09-30T01:59:59.553 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f597fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:59.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f597fa700 1 -- 172.21.15.7:0/4131827708 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54013470 con 0x7f8f5c096a50 2023-09-30T01:59:59.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f597fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:59.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f597fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:59.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f597fa700 10 monclient: dump: 2023-09-30T01:59:59.554 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:59.554 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:59.554 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.554 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.554 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:59.554 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.554 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:59.555 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:59.555 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:59.555 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f637fe700 10 monclient: _renew_subs 2023-09-30T01:59:59.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f637fe700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:59.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f5b7fe700 4 set_mon_vals no callback set 2023-09-30T01:59:59.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f637fe700 1 -- 172.21.15.7:0/4131827708 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f5c09f580 con 0x7f8f5c096a50 2023-09-30T01:59:59.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f597fa700 1 -- 172.21.15.7:0/4131827708 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f8f54013820 con 0x7f8f5c096a50 2023-09-30T01:59:59.555 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f637fe700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:59.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f637fe700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:59.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f637fe700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:59.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.535+0000 7f8f637fe700 1 -- 172.21.15.7:0/4131827708 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f5c064c10 con 0x7f8f5c096a50 2023-09-30T01:59:59.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.536+0000 7f8f597fa700 1 --2- 172.21.15.7:0/4131827708 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f8f480357e0 0x7f8f48037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.536+0000 7f8f597fa700 1 -- 172.21.15.7:0/4131827708 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f8f54048410 con 0x7f8f5c096a50 2023-09-30T01:59:59.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.536+0000 7f8f60d99700 1 --2- 172.21.15.7:0/4131827708 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f8f480357e0 0x7f8f48037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.536+0000 7f8f60d99700 10 monclient: get_auth_request con 0x7f8f480357e0 auth_method 0 2023-09-30T01:59:59.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.538+0000 7f8f60d99700 1 --2- 172.21.15.7:0/4131827708 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f8f480357e0 0x7f8f48037c90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c09f180 tx=0x7f8f50017000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:59.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.540+0000 7f8f597fa700 1 -- 172.21.15.7:0/4131827708 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8f5401b770 con 0x7f8f5c096a50 2023-09-30T01:59:59.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.540+0000 7f8f597fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:59.557 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.540+0000 7f8f597fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:59.735 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.735+0000 7f8f637fe700 1 -- 172.21.15.7:0/4131827708 --> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f5c0a1560 con 0x7f8f480357e0 2023-09-30T01:59:59.736 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.736+0000 7f8f597fa700 1 -- 172.21.15.7:0/4131827708 <== mgr.4105 v2:172.21.15.50:6824/86200 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8f50011b10 con 0x7f8f480357e0 2023-09-30T01:59:59.738 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.738+0000 7f8f637fe700 1 -- 172.21.15.7:0/4131827708 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f8f480357e0 msgr2=0x7f8f48037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.738+0000 7f8f637fe700 1 --2- 172.21.15.7:0/4131827708 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f8f480357e0 0x7f8f48037c90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c09f180 tx=0x7f8f50017000 comp rx=0 tx=0).stop 2023-09-30T01:59:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.738+0000 7f8f637fe700 10 monclient: shutdown 2023-09-30T01:59:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.738+0000 7f8f637fe700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.738+0000 7f8f637fe700 1 -- 172.21.15.7:0/4131827708 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f5c096a50 msgr2=0x7f8f5c12d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.738+0000 7f8f637fe700 1 --2- 172.21.15.7:0/4131827708 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f5c096a50 0x7f8f5c12d790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8f5400ec50 tx=0x7f8f54009250 comp rx=0 tx=0).stop 2023-09-30T01:59:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.738+0000 7f8f637fe700 1 -- 172.21.15.7:0/4131827708 shutdown_connections 2023-09-30T01:59:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.738+0000 7f8f637fe700 1 --2- 172.21.15.7:0/4131827708 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f8f480357e0 0x7f8f48037c90 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.738+0000 7f8f637fe700 1 --2- 172.21.15.7:0/4131827708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8f5c12ef00 0x7f8f5c09d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.739 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.738+0000 7f8f637fe700 1 --2- 172.21.15.7:0/4131827708 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f8f5c096a50 0x7f8f5c12d790 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.738+0000 7f8f637fe700 1 --2- 172.21.15.7:0/4131827708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f5c096e20 0x7f8f5c12dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.738+0000 7f8f637fe700 1 -- 172.21.15.7:0/4131827708 >> 172.21.15.7:0/4131827708 conn(0x7f8f5c070cf0 msgr2=0x7f8f5c130cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:59.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.738+0000 7f8f637fe700 1 -- 172.21.15.7:0/4131827708 shutdown_connections 2023-09-30T01:59:59.740 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.738+0000 7f8f637fe700 1 -- 172.21.15.7:0/4131827708 wait complete. 2023-09-30T01:59:59.740 INFO:teuthology.orchestra.run.smithi007.stderr:instructed osd(s) 5 to deep-scrub 2023-09-30T01:59:59.750 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T01:59:59.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.867+0000 7f9a6a130700 10 monclient: get_monmap_and_config 2023-09-30T01:59:59.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.867+0000 7f9a6a130700 10 monclient: build_initial_monmap 2023-09-30T01:59:59.867 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.867+0000 7f9a6a130700 10 monclient: monmap: 2023-09-30T01:59:59.867 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:59.868 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:59.868 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:59:59.868606+0000 2023-09-30T01:59:59.868 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:59:59.868606+0000 2023-09-30T01:59:59.868 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:59.868 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.868 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T01:59:59.868 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T01:59:59.868 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T01:59:59.868 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T01:59:59.868 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T01:59:59.868 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.868+0000 7f9a6a130700 1 Processor -- start 2023-09-30T01:59:59.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.868+0000 7f9a6a130700 1 -- start start 2023-09-30T01:59:59.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.868+0000 7f9a6a130700 10 monclient: init 2023-09-30T01:59:59.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.868+0000 7f9a6a130700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:59.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.868+0000 7f9a6a130700 10 monclient: _add_conns ranks=[0,1,2,3,4] 2023-09-30T01:59:59.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.868+0000 7f9a6a130700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a640a00f0 addr v1:172.21.15.7:6789/0 2023-09-30T01:59:59.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.868+0000 7f9a6a130700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9a64098e50 addr v1:172.21.15.50:6789/0 2023-09-30T01:59:59.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.868+0000 7f9a6a130700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a640a12e0 0x7f9a64137b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.868+0000 7f9a6a130700 10 monclient(hunting): picked mon.noname-b con 0x7f9a640a12e0 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:59.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.868+0000 7f9a6a130700 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a6409c740 con 0x7f9a640a00f0 2023-09-30T01:59:59.871 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.868+0000 7f9a6a130700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a64098a30 con 0x7f9a64098e50 2023-09-30T01:59:59.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.868+0000 7f9a6a130700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:59.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.868+0000 7f9a6a130700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a64098470 con 0x7f9a640a12e0 2023-09-30T01:59:59.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.868+0000 7f9a6a130700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:59.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.868+0000 7f9a6a130700 10 monclient(hunting): authenticate will time out at 1488.933972s 2023-09-30T01:59:59.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.869+0000 7f9a63fff700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a640a12e0 0x7f9a64137b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.869+0000 7f9a637fe700 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f9a640a00f0 0x7f9a640a04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48978/0 (socket says 172.21.15.7:48978) 2023-09-30T01:59:59.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.869+0000 7f9a637fe700 1 -- 172.21.15.7:0/3721317864 learned_addr learned my addr 172.21.15.7:0/3721317864 (peer_addr_for_me v1:172.21.15.7:0/0) 2023-09-30T01:59:59.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.869+0000 7f9a63fff700 10 monclient(hunting): get_auth_request con 0x7f9a640a12e0 auth_method 0 2023-09-30T01:59:59.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.869+0000 7f9a63fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:59.872 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.869+0000 7f9a63fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:59.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.869+0000 7f9a63fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:59.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.869+0000 7f9a63fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:59.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.869+0000 7f9a63fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:59.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.869+0000 7f9a63fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:59.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.869+0000 7f9a63fff700 10 monclient(hunting): handle_auth_done global_id 4838 payload 274 2023-09-30T01:59:59.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.869+0000 7f9a63fff700 1 -- 172.21.15.7:0/3721317864 >> v1:172.21.15.50:6789/0 conn(0x7f9a64098e50 legacy=0x7f9a640a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:59.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.869+0000 7f9a63fff700 1 -- 172.21.15.7:0/3721317864 >> v1:172.21.15.7:6789/0 conn(0x7f9a640a00f0 legacy=0x7f9a640a04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T01:59:59.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.869+0000 7f9a63fff700 10 monclient: _finish_hunting 0 2023-09-30T01:59:59.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.869+0000 7f9a63fff700 1 monclient: found mon.noname-b 2023-09-30T01:59:59.873 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.869+0000 7f9a63fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:59.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.869+0000 7f9a63fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T01:59:59.874 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.869+0000 7f9a63fff700 1 -- 172.21.15.7:0/3721317864 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a64098190 con 0x7f9a640a12e0 2023-09-30T01:59:59.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.871+0000 7f9a63fff700 1 -- 172.21.15.7:0/3721317864 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a640a12e0 msgr2=0x7f9a64137b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:59.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.871+0000 7f9a63fff700 1 --2- 172.21.15.7:0/3721317864 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a640a12e0 0x7f9a64137b70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009e90 tx=0x7f9a5c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=982b50b326d39605 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:59.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.871+0000 7f9a627fc700 1 -- 172.21.15.7:0/3721317864 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c00e070 con 0x7f9a640a12e0 2023-09-30T01:59:59.883 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.871+0000 7f9a627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:59.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.871+0000 7f9a627fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T01:59:59.884 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.871+0000 7f9a627fc700 10 monclient: dump: 2023-09-30T01:59:59.884 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:59.884 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:59.884 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.884 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.884 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:59.884 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.884 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:59.884 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:59.885 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:59.885 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.871+0000 7f9a627fc700 10 monclient: _finish_auth 0 2023-09-30T01:59:59.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.871+0000 7f9a627fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:59.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.871+0000 7f9a627fc700 1 -- 172.21.15.7:0/3721317864 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5c012430 con 0x7f9a640a12e0 2023-09-30T01:59:59.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.871+0000 7f9a6a130700 5 monclient: authenticate success, global_id 4838 2023-09-30T01:59:59.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.871+0000 7f9a6a130700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T01:59:59.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.871+0000 7f9a627fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:59.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.871+0000 7f9a6a130700 10 monclient: get_monmap_and_config success 2023-09-30T01:59:59.885 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.871+0000 7f9a627fc700 1 -- 172.21.15.7:0/3721317864 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c00bea0 con 0x7f9a640a12e0 2023-09-30T01:59:59.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.871+0000 7f9a627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:59.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.871+0000 7f9a6a130700 4 set_mon_vals no callback set 2023-09-30T01:59:59.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.871+0000 7f9a627fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:59.886 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.871+0000 7f9a627fc700 10 monclient: dump: 2023-09-30T01:59:59.886 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:59.886 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:59.886 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.886 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.886 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:59.886 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.887 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:59.887 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:59.887 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:59.887 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.872+0000 7f9a6a130700 10 monclient: shutdown 2023-09-30T01:59:59.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.872+0000 7f9a6a130700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T01:59:59.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.872+0000 7f9a6a130700 1 -- 172.21.15.7:0/3721317864 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a640a12e0 msgr2=0x7f9a64137b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:59.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.872+0000 7f9a6a130700 1 --2- 172.21.15.7:0/3721317864 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a640a12e0 0x7f9a64137b70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009e90 tx=0x7f9a5c009780 comp rx=0 tx=0).stop 2023-09-30T01:59:59.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.872+0000 7f9a6a130700 1 -- 172.21.15.7:0/3721317864 shutdown_connections 2023-09-30T01:59:59.887 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.872+0000 7f9a6a130700 1 --2- 172.21.15.7:0/3721317864 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a640a12e0 0x7f9a64137b70 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.872+0000 7f9a6a130700 1 -- 172.21.15.7:0/3721317864 >> 172.21.15.7:0/3721317864 conn(0x7f9a64070cf0 msgr2=0x7f9a640710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T01:59:59.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.872+0000 7f9a6a130700 1 -- 172.21.15.7:0/3721317864 shutdown_connections 2023-09-30T01:59:59.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.872+0000 7f9a6a130700 1 -- 172.21.15.7:0/3721317864 wait complete. 2023-09-30T01:59:59.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.872+0000 7f9a6a130700 10 monclient: build_initial_monmap 2023-09-30T01:59:59.888 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.872+0000 7f9a6a130700 10 monclient: monmap: 2023-09-30T01:59:59.888 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T01:59:59.888 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T01:59:59.888 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T01:59:59.888 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T01:59:59.888 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T01:59:59.888 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.889 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T01:59:59.889 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T01:59:59.889 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T01:59:59.889 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a6a130700 1 Processor -- start 2023-09-30T01:59:59.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a6a130700 1 -- start start 2023-09-30T01:59:59.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a6a130700 10 monclient: init 2023-09-30T01:59:59.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a6a130700 10 monclient: _reopen_session rank -1 2023-09-30T01:59:59.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a6a130700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T01:59:59.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a6a130700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a64098e50 0x7f9a6413e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.889 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a6a130700 10 monclient(hunting): picked mon.noname-c con 0x7f9a64098e50 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T01:59:59.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a6a130700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a640a00f0 0x7f9a6413dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a6a130700 10 monclient(hunting): picked mon.noname-b con 0x7f9a640a00f0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T01:59:59.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a6a130700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a640a12e0 0x7f9a64141b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a6a130700 10 monclient(hunting): picked mon.noname-a con 0x7f9a640a12e0 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T01:59:59.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a6a130700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:59.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a6a130700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a64122ff0 con 0x7f9a640a12e0 2023-09-30T01:59:59.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a6a130700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:59.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a6a130700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a641212e0 con 0x7f9a640a00f0 2023-09-30T01:59:59.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a6a130700 10 monclient(hunting): start opening mon connection 2023-09-30T01:59:59.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a6a130700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6413f100 con 0x7f9a64098e50 2023-09-30T01:59:59.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a6a130700 10 monclient(hunting): _renew_subs 2023-09-30T01:59:59.890 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a6a130700 10 monclient(hunting): authenticate will time out at 1488.938825s 2023-09-30T01:59:59.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a637fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a64098e50 0x7f9a6413e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a63fff700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a640a12e0 0x7f9a64141b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a637fe700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a64098e50 0x7f9a6413e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:41506/0 (socket says 172.21.15.7:41506) 2023-09-30T01:59:59.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a637fe700 1 -- 172.21.15.7:0/3281715121 learned_addr learned my addr 172.21.15.7:0/3281715121 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T01:59:59.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a637fe700 10 monclient(hunting): get_auth_request con 0x7f9a64098e50 auth_method 0 2023-09-30T01:59:59.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:59.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a637fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:59.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a637fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:59.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a63fff700 10 monclient(hunting): get_auth_request con 0x7f9a640a12e0 auth_method 0 2023-09-30T01:59:59.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a63fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:59.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a63fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:59.891 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.873+0000 7f9a63fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:59.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a62ffd700 1 --2- 172.21.15.7:0/3281715121 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a640a00f0 0x7f9a6413dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:59.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:59.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:59.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a62ffd700 10 monclient(hunting): get_auth_request con 0x7f9a640a00f0 auth_method 0 2023-09-30T01:59:59.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a62ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T01:59:59.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a62ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T01:59:59.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a62ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T01:59:59.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a63fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:59.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a63fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:59.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a63fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:59.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a62ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T01:59:59.892 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a62ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T01:59:59.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a62ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T01:59:59.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a637fe700 10 monclient(hunting): handle_auth_done global_id 4841 payload 824 2023-09-30T01:59:59.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a637fe700 1 -- 172.21.15.7:0/3281715121 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a640a00f0 msgr2=0x7f9a6413dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:59.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a637fe700 1 --2- 172.21.15.7:0/3281715121 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a640a00f0 0x7f9a6413dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a637fe700 1 -- 172.21.15.7:0/3281715121 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a640a12e0 msgr2=0x7f9a64141b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T01:59:59.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a637fe700 1 --2- 172.21.15.7:0/3281715121 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a640a12e0 0x7f9a64141b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T01:59:59.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a637fe700 10 monclient: _finish_hunting 0 2023-09-30T01:59:59.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a637fe700 1 monclient: found mon.noname-c 2023-09-30T01:59:59.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T01:59:59.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a637fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T01:59:59.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a637fe700 1 -- 172.21.15.7:0/3281715121 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c009a20 con 0x7f9a64098e50 2023-09-30T01:59:59.893 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.874+0000 7f9a63fff700 1 --2- 172.21.15.7:0/3281715121 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a640a12e0 0x7f9a64141b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T01:59:59.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a637fe700 1 -- 172.21.15.7:0/3281715121 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a64098e50 msgr2=0x7f9a6413e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T01:59:59.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a637fe700 1 --2- 172.21.15.7:0/3281715121 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a64098e50 0x7f9a6413e5c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400c5e0 tx=0x7f9a54004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:59.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a4b7fe700 1 -- 172.21.15.7:0/3281715121 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a54005d40 con 0x7f9a64098e50 2023-09-30T01:59:59.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a4b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:59.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a4b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T01:59:59.894 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a4b7fe700 10 monclient: dump: 2023-09-30T01:59:59.894 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:59.894 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:59.894 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.894 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.894 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:59.895 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.895 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:59.895 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:59.895 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:59.895 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a4b7fe700 10 monclient: _finish_auth 0 2023-09-30T01:59:59.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a4b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T01:59:59.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a4b7fe700 1 -- 172.21.15.7:0/3281715121 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a540070d0 con 0x7f9a64098e50 2023-09-30T01:59:59.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a6a130700 5 monclient: authenticate success, global_id 4841 2023-09-30T01:59:59.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a6a130700 10 monclient: _renew_subs 2023-09-30T01:59:59.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a6a130700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:59.895 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a6a130700 1 -- 172.21.15.7:0/3281715121 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a64122d10 con 0x7f9a64098e50 2023-09-30T01:59:59.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a4b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T01:59:59.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a4b7fe700 1 -- 172.21.15.7:0/3281715121 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a54007490 con 0x7f9a64098e50 2023-09-30T01:59:59.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a4b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T01:59:59.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a4b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T01:59:59.896 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a4b7fe700 10 monclient: dump: 2023-09-30T01:59:59.896 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T01:59:59.896 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T01:59:59.896 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.896 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T01:59:59.896 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T01:59:59.896 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T01:59:59.896 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T01:59:59.896 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T01:59:59.897 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T01:59:59.897 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T01:59:59.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a6a130700 10 monclient: _renew_subs 2023-09-30T01:59:59.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a6a130700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:59.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a61ffb700 4 set_mon_vals no callback set 2023-09-30T01:59:59.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.876+0000 7f9a6a130700 1 -- 172.21.15.7:0/3281715121 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a64122000 con 0x7f9a64098e50 2023-09-30T01:59:59.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.877+0000 7f9a4b7fe700 1 -- 172.21.15.7:0/3281715121 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f9a5401b3b0 con 0x7f9a64098e50 2023-09-30T01:59:59.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.877+0000 7f9a6a130700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:59.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.877+0000 7f9a6a130700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:59.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.877+0000 7f9a6a130700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T01:59:59.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.877+0000 7f9a6a130700 1 -- 172.21.15.7:0/3281715121 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a64123620 con 0x7f9a64098e50 2023-09-30T01:59:59.897 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.877+0000 7f9a4b7fe700 1 --2- 172.21.15.7:0/3281715121 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9a500357e0 0x7f9a50037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T01:59:59.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.877+0000 7f9a4b7fe700 1 -- 172.21.15.7:0/3281715121 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f9a54047130 con 0x7f9a64098e50 2023-09-30T01:59:59.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.878+0000 7f9a62ffd700 1 --2- 172.21.15.7:0/3281715121 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9a500357e0 0x7f9a50037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T01:59:59.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.878+0000 7f9a62ffd700 10 monclient: get_auth_request con 0x7f9a500357e0 auth_method 0 2023-09-30T01:59:59.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.880+0000 7f9a62ffd700 1 --2- 172.21.15.7:0/3281715121 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9a500357e0 0x7f9a50037c90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9a58003a30 tx=0x7f9a58007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T01:59:59.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.882+0000 7f9a4b7fe700 1 -- 172.21.15.7:0/3281715121 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9a5400f670 con 0x7f9a64098e50 2023-09-30T01:59:59.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.882+0000 7f9a4b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T01:59:59.898 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T01:59:59.882+0000 7f9a4b7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T01:59:59.999 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-09-30T01:59:59.999+0000 7fb5b14f1700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T02:00:00.069 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:00.069+0000 7f9a6a130700 1 -- 172.21.15.7:0/3281715121 --> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9a64123160 con 0x7f9a500357e0 2023-09-30T02:00:00.072 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:00.072+0000 7f9a4b7fe700 1 -- 172.21.15.7:0/3281715121 <== mgr.4105 v2:172.21.15.50:6824/86200 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53667 (secure 0 0 0) 0x7f9a64123160 con 0x7f9a500357e0 2023-09-30T02:00:00.073 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T02:00:00.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:00.074+0000 7f9a6a130700 1 -- 172.21.15.7:0/3281715121 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9a500357e0 msgr2=0x7f9a50037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:00.074 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:00.074+0000 7f9a6a130700 1 --2- 172.21.15.7:0/3281715121 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9a500357e0 0x7f9a50037c90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9a58003a30 tx=0x7f9a58007ea0 comp rx=0 tx=0).stop 2023-09-30T02:00:00.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:00.074+0000 7f9a6a130700 10 monclient: shutdown 2023-09-30T02:00:00.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:00.074+0000 7f9a6a130700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:00:00.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:00.074+0000 7f9a6a130700 1 -- 172.21.15.7:0/3281715121 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a64098e50 msgr2=0x7f9a6413e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:00.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:00.074+0000 7f9a6a130700 1 --2- 172.21.15.7:0/3281715121 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a64098e50 0x7f9a6413e5c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400c5e0 tx=0x7f9a54004a80 comp rx=0 tx=0).stop 2023-09-30T02:00:00.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:00.074+0000 7f9a6a130700 1 -- 172.21.15.7:0/3281715121 shutdown_connections 2023-09-30T02:00:00.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:00.074+0000 7f9a6a130700 1 --2- 172.21.15.7:0/3281715121 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f9a500357e0 0x7f9a50037c90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:00.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:00.074+0000 7f9a6a130700 1 --2- 172.21.15.7:0/3281715121 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a640a12e0 0x7f9a64141b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:00.075 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:00.074+0000 7f9a6a130700 1 --2- 172.21.15.7:0/3281715121 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f9a64098e50 0x7f9a6413e5c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:00.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:00.074+0000 7f9a6a130700 1 --2- 172.21.15.7:0/3281715121 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9a640a00f0 0x7f9a6413dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:00.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:00.074+0000 7f9a6a130700 1 -- 172.21.15.7:0/3281715121 >> 172.21.15.7:0/3281715121 conn(0x7f9a64070cf0 msgr2=0x7f9a640999b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:00.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:00.074+0000 7f9a6a130700 1 -- 172.21.15.7:0/3281715121 shutdown_connections 2023-09-30T02:00:00.076 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:00.074+0000 7f9a6a130700 1 -- 172.21.15.7:0/3281715121 wait complete. 2023-09-30T02:00:00.076 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-09-30T02:00:00.086 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2023-09-30T01:59:58.329570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":32644,"kb_used_data":1876,"kb_used_omap":0,"kb_used_meta":30720,"kb_avail":566198396,"statfs":{"total":579820584960,"available":579787157504,"internally_reserved":0,"allocated":1921024,"data_stored":1316285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31457280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002444"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.042899+0000","last_change":"2023-09-30T01:54:47.042422+0000","last_active":"2023-09-30T01:54:47.042899+0000","last_peered":"2023-09-30T01:54:47.042899+0000","last_clean":"2023-09-30T01:54:47.042899+0000","last_became_active":"2023-09-30T01:54:47.042183+0000","last_became_peered":"2023-09-30T01:54:47.042183+0000","last_unstale":"2023-09-30T01:54:47.042899+0000","last_undegraded":"2023-09-30T01:54:47.042899+0000","last_fullsized":"2023-09-30T01:54:47.042899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:14:57.725898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.897849+0000","last_change":"2023-09-30T01:54:46.896909+0000","last_active":"2023-09-30T01:54:46.897849+0000","last_peered":"2023-09-30T01:54:46.897849+0000","last_clean":"2023-09-30T01:54:46.897849+0000","last_became_active":"2023-09-30T01:54:46.896620+0000","last_became_peered":"2023-09-30T01:54:46.896620+0000","last_unstale":"2023-09-30T01:54:46.897849+0000","last_undegraded":"2023-09-30T01:54:46.897849+0000","last_fullsized":"2023-09-30T01:54:46.897849+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:28:25.053882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.042370+0000","last_change":"2023-09-30T01:54:47.041973+0000","last_active":"2023-09-30T01:54:47.042370+0000","last_peered":"2023-09-30T01:54:47.042370+0000","last_clean":"2023-09-30T01:54:47.042370+0000","last_became_active":"2023-09-30T01:54:47.041671+0000","last_became_peered":"2023-09-30T01:54:47.041671+0000","last_unstale":"2023-09-30T01:54:47.042370+0000","last_undegraded":"2023-09-30T01:54:47.042370+0000","last_fullsized":"2023-09-30T01:54:47.042370+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:08:29.771169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.896726+0000","last_change":"2023-09-30T01:54:46.896432+0000","last_active":"2023-09-30T01:54:46.896726+0000","last_peered":"2023-09-30T01:54:46.896726+0000","last_clean":"2023-09-30T01:54:46.896726+0000","last_became_active":"2023-09-30T01:54:46.896191+0000","last_became_peered":"2023-09-30T01:54:46.896191+0000","last_unstale":"2023-09-30T01:54:46.896726+0000","last_undegraded":"2023-09-30T01:54:46.896726+0000","last_fullsized":"2023-09-30T01:54:46.896726+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:36:51.492282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":160,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.202781+0000","last_change":"2023-09-30T01:54:45.896629+0000","last_active":"2023-09-30T01:54:47.202781+0000","last_peered":"2023-09-30T01:54:47.202781+0000","last_clean":"2023-09-30T01:54:47.202781+0000","last_became_active":"2023-09-30T01:54:45.896315+0000","last_became_peered":"2023-09-30T01:54:45.896315+0000","last_unstale":"2023-09-30T01:54:47.202781+0000","last_undegraded":"2023-09-30T01:54:47.202781+0000","last_fullsized":"2023-09-30T01:54:47.202781+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:44.377605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:44.377605+0000","last_clean_scrub_stamp":"2023-09-30T01:54:44.377605+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T13:48:02.317525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.899133+0000","last_change":"2023-09-30T01:54:46.898655+0000","last_active":"2023-09-30T01:54:46.899133+0000","last_peered":"2023-09-30T01:54:46.899133+0000","last_clean":"2023-09-30T01:54:46.899133+0000","last_became_active":"2023-09-30T01:54:46.898402+0000","last_became_peered":"2023-09-30T01:54:46.898402+0000","last_unstale":"2023-09-30T01:54:46.899133+0000","last_undegraded":"2023-09-30T01:54:46.899133+0000","last_fullsized":"2023-09-30T01:54:46.899133+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:07:41.886018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.897962+0000","last_change":"2023-09-30T01:54:46.896375+0000","last_active":"2023-09-30T01:54:46.897962+0000","last_peered":"2023-09-30T01:54:46.897962+0000","last_clean":"2023-09-30T01:54:46.897962+0000","last_became_active":"2023-09-30T01:54:46.895973+0000","last_became_peered":"2023-09-30T01:54:46.895973+0000","last_unstale":"2023-09-30T01:54:46.897962+0000","last_undegraded":"2023-09-30T01:54:46.897962+0000","last_fullsized":"2023-09-30T01:54:46.897962+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:52:22.627216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:46.898698+0000","last_change":"2023-09-30T01:54:46.898322+0000","last_active":"2023-09-30T01:54:46.898698+0000","last_peered":"2023-09-30T01:54:46.898698+0000","last_clean":"2023-09-30T01:54:46.898698+0000","last_became_active":"2023-09-30T01:54:46.898101+0000","last_became_peered":"2023-09-30T01:54:46.898101+0000","last_unstale":"2023-09-30T01:54:46.898698+0000","last_undegraded":"2023-09-30T01:54:46.898698+0000","last_fullsized":"2023-09-30T01:54:46.898698+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:45:31.230458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-09-30T01:54:47.042677+0000","last_change":"2023-09-30T01:54:47.042174+0000","last_active":"2023-09-30T01:54:47.042677+0000","last_peered":"2023-09-30T01:54:47.042677+0000","last_clean":"2023-09-30T01:54:47.042677+0000","last_became_active":"2023-09-30T01:54:47.041914+0000","last_became_peered":"2023-09-30T01:54:47.041914+0000","last_unstale":"2023-09-30T01:54:47.042677+0000","last_undegraded":"2023-09-30T01:54:47.042677+0000","last_fullsized":"2023-09-30T01:54:47.042677+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:54:45.888656+0000","last_clean_scrub_stamp":"2023-09-30T01:54:45.888656+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:16:46.356909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640322,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5076,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366764,"statfs":{"total":96636764160,"available":96631566336,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52500000000000002,"15min":0.54300000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.69199999999999995,"5min":0.747,"15min":0.747},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53500000000000003,"15min":0.56100000000000005},"min":{"1min":0.40799999999999997,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66400000000000003,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.40000000000000002}]},{"osd":1,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.51500000000000001,"15min":0.53700000000000003},"min":{"1min":0.376,"5min":0.313,"15min":0.313},"max":{"1min":0.66400000000000003,"5min":0.751,"15min":0.751},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.53500000000000003,"15min":0.56100000000000005},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.629,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":1.4039999999999999}]},{"osd":2,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55100000000000005,"15min":0.57299999999999995},"min":{"1min":0.42499999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.73899999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55300000000000005,"15min":0.56499999999999995},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72599999999999998,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.45200000000000001}]},{"osd":3,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41999999999999998,"15min":0.443},"min":{"1min":0.29899999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.60499999999999998,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.62},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45500000000000002,"15min":0.47899999999999998},"min":{"1min":0.312,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.432}]},{"osd":4,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.45500000000000002,"15min":0.48999999999999999},"min":{"1min":0.311,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.498,"5min":0.50900000000000001,"15min":0.53500000000000003},"min":{"1min":0.36599999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.76700000000000002,"5min":0.874,"15min":0.874},"last":0.59599999999999997}]}]},{"osd":4,"up_from":11,"seq":47244640320,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5076,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366764,"statfs":{"total":96636764160,"available":96631566336,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51000000000000001,"15min":0.51400000000000001},"min":{"1min":0.35399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70199999999999996,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.439},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.51600000000000001,"15min":0.48799999999999999},"min":{"1min":0.38800000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.68799999999999994,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.32300000000000001}]},{"osd":1,"last update":"Sat Sep 30 01:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51700000000000002,"15min":0.49099999999999999},"min":{"1min":0.39500000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.65100000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53500000000000003,"15min":0.52500000000000002},"min":{"1min":0.40200000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.65600000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.36899999999999999}]},{"osd":2,"last update":"Sat Sep 30 01:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.50600000000000001},"min":{"1min":0.32900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.72799999999999998,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55200000000000005,"15min":0.52600000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.76100000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.5}]},{"osd":3,"last update":"Sat Sep 30 01:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40799999999999997,"15min":0.40999999999999998},"min":{"1min":0.28299999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.432,"5min":0.44800000000000001,"15min":0.45500000000000002},"min":{"1min":0.33200000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.624,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.45700000000000002}]},{"osd":5,"last update":"Sat Sep 30 01:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.45100000000000001,"15min":0.45500000000000002},"min":{"1min":0.31,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.67100000000000004,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.48699999999999999,"15min":0.501},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.68899999999999995,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.48299999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640320,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6108,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365732,"statfs":{"total":96636764160,"available":96630509568,"internally_reserved":0,"allocated":675840,"data_stored":572143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.36799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.70199999999999996,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.53800000000000003,"15min":0.54100000000000004},"min":{"1min":0.35899999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.71499999999999997,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.48399999999999999}]},{"osd":1,"last update":"Sat Sep 30 01:58:53 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51400000000000001,"15min":0.51200000000000001},"min":{"1min":0.36599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.50600000000000001,"15min":0.497},"min":{"1min":0.38400000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.63500000000000001,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.38500000000000001}]},{"osd":2,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.51500000000000001,"15min":0.51100000000000001},"min":{"1min":0.434,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.64100000000000001,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.37},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54000000000000004,"15min":0.54800000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69299999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.46400000000000002}]},{"osd":4,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.26900000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.59499999999999997,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.48899999999999999,"15min":0.501},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.60599999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.435}]},{"osd":5,"last update":"Sat Sep 30 01:58:50 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.45800000000000002,"15min":0.47399999999999998},"min":{"1min":0.28599999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.65400000000000003,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.441,"5min":0.46999999999999997,"15min":0.48199999999999998},"min":{"1min":0.31900000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.59999999999999998,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.30399999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5140,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366700,"statfs":{"total":96636764160,"available":96631500800,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35299999999999998},"min":{"1min":0.245,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.47999999999999998,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.39300000000000002,"15min":0.38500000000000001},"min":{"1min":0.28100000000000003,"5min":0.254,"15min":0.254},"max":{"1min":0.54300000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.39500000000000002}]},{"osd":1,"last update":"Sat Sep 30 01:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.36799999999999999,"15min":0.37},"min":{"1min":0.23999999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.55800000000000005,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.374,"15min":0.36199999999999999},"min":{"1min":0.251,"5min":0.222,"15min":0.222},"max":{"1min":0.56799999999999995,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.35299999999999998}]},{"osd":3,"last update":"Sat Sep 30 01:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57399999999999995,"15min":0.56999999999999995},"min":{"1min":0.43099999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.56599999999999995},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.58899999999999997}]},{"osd":4,"last update":"Sat Sep 30 01:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.436,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.82699999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59599999999999997,"15min":0.58399999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.63300000000000001}]},{"osd":5,"last update":"Sat Sep 30 01:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57999999999999996,"15min":0.56999999999999995},"min":{"1min":0.44700000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61499999999999999,"15min":0.61299999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.93799999999999994,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.64600000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640320,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6104,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365736,"statfs":{"total":96636764160,"available":96630513664,"internally_reserved":0,"allocated":671744,"data_stored":571282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 01:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.34100000000000003,"15min":0.33400000000000002},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.48799999999999999,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.377,"5min":0.39100000000000001,"15min":0.39600000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.51000000000000001,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.34100000000000003}]},{"osd":2,"last update":"Sat Sep 30 01:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.35699999999999998,"15min":0.35999999999999999},"min":{"1min":0.21299999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.53200000000000003,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.41599999999999998,"15min":0.42599999999999999},"min":{"1min":0.28299999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.54000000000000004,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.377}]},{"osd":3,"last update":"Sat Sep 30 01:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39100000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.75,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.55800000000000005,"15min":0.56000000000000005},"min":{"1min":0.39900000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.76900000000000002,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.76900000000000002}]},{"osd":4,"last update":"Sat Sep 30 01:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57599999999999996,"15min":0.57899999999999996},"min":{"1min":0.33400000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.72199999999999998,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56899999999999995,"15min":0.57599999999999996},"min":{"1min":0.34899999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73499999999999999,"5min":1.026,"15min":1.026},"last":0.69499999999999995}]},{"osd":5,"last update":"Sat Sep 30 01:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56599999999999995,"15min":0.57599999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76600000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.58699999999999997,"15min":0.58899999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.71199999999999997,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.70899999999999996}]}]},{"osd":1,"up_from":11,"seq":47244640322,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5140,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366700,"statfs":{"total":96636764160,"available":96631500800,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.34000000000000002,"15min":0.312},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.52900000000000003,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.34699999999999998,"15min":0.32200000000000001},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.50900000000000001,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.28699999999999998}]},{"osd":2,"last update":"Sat Sep 30 01:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.36099999999999999,"15min":0.33800000000000002},"min":{"1min":0.224,"5min":0.22,"15min":0.22},"max":{"1min":0.57799999999999996,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.313},{"interface":"front","average":{"1min":0.377,"5min":0.375,"15min":0.36399999999999999},"min":{"1min":0.27100000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.55600000000000005,"5min":0.62,"15min":0.62},"last":0.25600000000000001}]},{"osd":3,"last update":"Sat Sep 30 01:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.54400000000000004,"15min":0.52000000000000002},"min":{"1min":0.46000000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.373,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73299999999999998,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46999999999999997}]},{"osd":4,"last update":"Sat Sep 30 01:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57599999999999996,"15min":0.56399999999999995},"min":{"1min":0.40000000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.751,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.441,"5min":0.371,"15min":0.371},"max":{"1min":0.72899999999999998,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.41299999999999998}]},{"osd":5,"last update":"Sat Sep 30 01:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57299999999999995,"15min":0.55600000000000005},"min":{"1min":0.42799999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59299999999999997,"15min":0.57199999999999995},"min":{"1min":0.47599999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80800000000000005,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.5}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:00:00.090 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-09-30T01:54:45.888656+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=54, tm_sec=45, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=59, tm_sec=55, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:00:00.090 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-09-30T01:54:45.888656+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=54, tm_sec=45, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=59, tm_sec=55, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:00:00.090 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-09-30T01:54:45.888656+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=54, tm_sec=45, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=59, tm_sec=55, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:00:00.091 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-09-30T01:54:45.888656+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=54, tm_sec=45, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=59, tm_sec=55, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:00:00.091 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-09-30T01:54:44.377605+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=54, tm_sec=44, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=59, tm_sec=55, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:00:00.091 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-09-30T01:54:45.888656+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=54, tm_sec=45, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=59, tm_sec=55, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:00:00.091 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-09-30T01:54:45.888656+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=54, tm_sec=45, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=59, tm_sec=55, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:00:00.091 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-09-30T01:54:45.888656+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=54, tm_sec=45, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=59, tm_sec=55, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:00:00.091 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-09-30T01:54:45.888656+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=54, tm_sec=45, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=1, tm_min=59, tm_sec=55, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:00:00.091 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-30T02:00:20.093 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T02:00:20.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.226+0000 7f95e7337700 10 monclient: get_monmap_and_config 2023-09-30T02:00:20.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.226+0000 7f95e7337700 10 monclient: build_initial_monmap 2023-09-30T02:00:20.227 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.226+0000 7f95e7337700 10 monclient: monmap: 2023-09-30T02:00:20.227 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T02:00:20.227 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T02:00:20.227 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T02:00:20.227849+0000 2023-09-30T02:00:20.228 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T02:00:20.227849+0000 2023-09-30T02:00:20.228 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T02:00:20.228 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T02:00:20.228 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T02:00:20.228 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T02:00:20.228 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T02:00:20.228 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T02:00:20.228 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T02:00:20.228 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T02:00:20.228 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.227+0000 7f95e7337700 1 Processor -- start 2023-09-30T02:00:20.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.227+0000 7f95e7337700 1 -- start start 2023-09-30T02:00:20.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.227+0000 7f95e7337700 10 monclient: init 2023-09-30T02:00:20.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e7337700 10 monclient: _reopen_session rank -1 2023-09-30T02:00:20.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e7337700 10 monclient: _add_conns ranks=[3,2,1,0,4] 2023-09-30T02:00:20.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e7337700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f95e009a900 0x7f95e006f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:20.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e7337700 10 monclient(hunting): picked mon.noname-a con 0x7f95e009a900 addr v2:172.21.15.7:3300/0 2023-09-30T02:00:20.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e7337700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95e0130440 0x7f95e006f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:20.235 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e7337700 10 monclient(hunting): picked mon.noname-b con 0x7f95e0130440 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T02:00:20.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e7337700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f95e009bc40 addr v1:172.21.15.50:6789/0 2023-09-30T02:00:20.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e7337700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95e009ce10 con 0x7f95e009bc40 2023-09-30T02:00:20.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e7337700 10 monclient(hunting): start opening mon connection 2023-09-30T02:00:20.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e7337700 1 -- --> v2:172.21.15.7:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95e009cc10 con 0x7f95e009a900 2023-09-30T02:00:20.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e7337700 10 monclient(hunting): start opening mon connection 2023-09-30T02:00:20.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e7337700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95e009d3d0 con 0x7f95e0130440 2023-09-30T02:00:20.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e7337700 10 monclient(hunting): _renew_subs 2023-09-30T02:00:20.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e7337700 10 monclient(hunting): authenticate will time out at 1509.293174s 2023-09-30T02:00:20.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e48d2700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95e0130440 0x7f95e006f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:20.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e50d3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f95e009a900 0x7f95e006f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:20.236 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e50d3700 1 --2- >> v2:172.21.15.7:3300/0 conn(0x7f95e009a900 0x7f95e006f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36758/0 (socket says 172.21.15.7:36758) 2023-09-30T02:00:20.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e48d2700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95e0130440 0x7f95e006f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:47848/0 (socket says 172.21.15.7:47848) 2023-09-30T02:00:20.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e50d3700 1 -- 172.21.15.7:0/2553578106 learned_addr learned my addr 172.21.15.7:0/2553578106 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T02:00:20.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e50d3700 10 monclient(hunting): get_auth_request con 0x7f95e009a900 auth_method 0 2023-09-30T02:00:20.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e50d3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:00:20.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e50d3700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:00:20.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e50d3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:00:20.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e48d2700 10 monclient(hunting): get_auth_request con 0x7f95e0130440 auth_method 0 2023-09-30T02:00:20.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e48d2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:00:20.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e48d2700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:00:20.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e48d2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:00:20.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e50d3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:00:20.237 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e50d3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:00:20.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e50d3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:00:20.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e48d2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:00:20.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e48d2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:00:20.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e48d2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:00:20.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e50d3700 10 monclient(hunting): handle_auth_done global_id 4818 payload 274 2023-09-30T02:00:20.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e50d3700 1 -- 172.21.15.7:0/2553578106 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95e0130440 msgr2=0x7f95e006f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:20.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e50d3700 1 --2- 172.21.15.7:0/2553578106 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95e0130440 0x7f95e006f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:20.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e50d3700 1 -- 172.21.15.7:0/2553578106 >> v1:172.21.15.50:6789/0 conn(0x7f95e009bc40 legacy=0x7f95e0096e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:20.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e50d3700 10 monclient: _finish_hunting 0 2023-09-30T02:00:20.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e50d3700 1 monclient: found mon.noname-a 2023-09-30T02:00:20.238 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e50d3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:00:20.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e50d3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T02:00:20.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e50d3700 1 -- 172.21.15.7:0/2553578106 --> v2:172.21.15.7:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e009acd0 con 0x7f95e009a900 2023-09-30T02:00:20.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.228+0000 7f95e48d2700 1 --2- 172.21.15.7:0/2553578106 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95e0130440 0x7f95e006f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:00:20.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.230+0000 7f95e50d3700 1 -- 172.21.15.7:0/2553578106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95e009a900 msgr2=0x7f95e006f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T02:00:20.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.230+0000 7f95e50d3700 1 --2- 172.21.15.7:0/2553578106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95e009a900 0x7f95e006f2c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f95dc0099b0 tx=0x7f95dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e6ee3550a62d816 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:20.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95d7fff700 1 -- 172.21.15.7:0/2553578106 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95dc009ea0 con 0x7f95e009a900 2023-09-30T02:00:20.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:00:20.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95d7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:00:20.239 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95d7fff700 10 monclient: dump: 2023-09-30T02:00:20.239 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T02:00:20.239 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T02:00:20.239 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T02:00:20.240 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T02:00:20.240 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T02:00:20.240 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T02:00:20.240 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T02:00:20.240 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T02:00:20.240 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T02:00:20.240 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T02:00:20.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95d7fff700 10 monclient: _finish_auth 0 2023-09-30T02:00:20.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95d7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:00:20.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95e7337700 5 monclient: authenticate success, global_id 4818 2023-09-30T02:00:20.240 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95d7fff700 1 -- 172.21.15.7:0/2553578106 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95dc00ba30 con 0x7f95e009a900 2023-09-30T02:00:20.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95d7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:00:20.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95e7337700 10 monclient: get_monmap_and_config success 2023-09-30T02:00:20.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95e7337700 4 set_mon_vals no callback set 2023-09-30T02:00:20.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95d7fff700 1 -- 172.21.15.7:0/2553578106 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95dc00bdf0 con 0x7f95e009a900 2023-09-30T02:00:20.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:00:20.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95d7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:00:20.241 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95d7fff700 10 monclient: dump: 2023-09-30T02:00:20.241 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T02:00:20.242 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T02:00:20.242 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T02:00:20.242 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T02:00:20.242 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T02:00:20.242 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T02:00:20.242 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T02:00:20.242 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T02:00:20.242 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T02:00:20.242 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T02:00:20.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95e7337700 10 monclient: shutdown 2023-09-30T02:00:20.242 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95e7337700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:00:20.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95e7337700 1 -- 172.21.15.7:0/2553578106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95e009a900 msgr2=0x7f95e006f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:20.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95e7337700 1 --2- 172.21.15.7:0/2553578106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95e009a900 0x7f95e006f2c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f95dc0099b0 tx=0x7f95dc009250 comp rx=0 tx=0).stop 2023-09-30T02:00:20.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95e7337700 1 -- 172.21.15.7:0/2553578106 shutdown_connections 2023-09-30T02:00:20.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95e7337700 1 --2- 172.21.15.7:0/2553578106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95e009a900 0x7f95e006f2c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:20.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95e7337700 1 --2- 172.21.15.7:0/2553578106 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95e0130440 0x7f95e006f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:20.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95e7337700 1 -- 172.21.15.7:0/2553578106 >> 172.21.15.7:0/2553578106 conn(0x7f95e009e0b0 msgr2=0x7f95e009e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:20.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95e7337700 1 -- 172.21.15.7:0/2553578106 shutdown_connections 2023-09-30T02:00:20.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.231+0000 7f95e7337700 1 -- 172.21.15.7:0/2553578106 wait complete. 2023-09-30T02:00:20.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.232+0000 7f95e7337700 10 monclient: build_initial_monmap 2023-09-30T02:00:20.243 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.232+0000 7f95e7337700 10 monclient: monmap: 2023-09-30T02:00:20.244 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T02:00:20.244 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:00:20.244 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T02:00:20.244 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T02:00:20.244 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T02:00:20.244 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T02:00:20.244 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T02:00:20.244 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T02:00:20.244 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T02:00:20.244 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T02:00:20.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.232+0000 7f95e7337700 1 Processor -- start 2023-09-30T02:00:20.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.232+0000 7f95e7337700 1 -- start start 2023-09-30T02:00:20.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.232+0000 7f95e7337700 10 monclient: init 2023-09-30T02:00:20.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e7337700 10 monclient: _reopen_session rank -1 2023-09-30T02:00:20.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e7337700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:00:20.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e7337700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95e009a900 0x7f95e0119de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:20.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e7337700 10 monclient(hunting): picked mon.noname-c con 0x7f95e009a900 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T02:00:20.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e7337700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95e009bc40 0x7f95e011a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:20.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e7337700 10 monclient(hunting): picked mon.noname-a con 0x7f95e009bc40 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T02:00:20.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e7337700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95e0130440 0x7f95e011a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:20.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e7337700 10 monclient(hunting): picked mon.noname-b con 0x7f95e0130440 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T02:00:20.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e7337700 10 monclient(hunting): start opening mon connection 2023-09-30T02:00:20.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e7337700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e0130aa0 con 0x7f95e009bc40 2023-09-30T02:00:20.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e7337700 10 monclient(hunting): start opening mon connection 2023-09-30T02:00:20.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e7337700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e012e250 con 0x7f95e0130440 2023-09-30T02:00:20.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e7337700 10 monclient(hunting): start opening mon connection 2023-09-30T02:00:20.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e7337700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95e011d7d0 con 0x7f95e009a900 2023-09-30T02:00:20.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e7337700 10 monclient(hunting): _renew_subs 2023-09-30T02:00:20.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e7337700 10 monclient(hunting): authenticate will time out at 1509.298221s 2023-09-30T02:00:20.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e50d3700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95e009a900 0x7f95e0119de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:20.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e48d2700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95e009bc40 0x7f95e011a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:20.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e50d3700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95e009a900 0x7f95e0119de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:47850/0 (socket says 172.21.15.7:47850) 2023-09-30T02:00:20.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e50d3700 1 -- 172.21.15.7:0/2625629839 learned_addr learned my addr 172.21.15.7:0/2625629839 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T02:00:20.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e50d3700 10 monclient(hunting): get_auth_request con 0x7f95e009a900 auth_method 0 2023-09-30T02:00:20.246 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e50d3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:00:20.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e50d3700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:00:20.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e50d3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:00:20.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e48d2700 10 monclient(hunting): get_auth_request con 0x7f95e009bc40 auth_method 0 2023-09-30T02:00:20.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e48d2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:00:20.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e48d2700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:00:20.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e48d2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:00:20.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e50d3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:00:20.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e50d3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:00:20.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e50d3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:00:20.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e48d2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:00:20.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e48d2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:00:20.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e48d2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:00:20.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e58d4700 1 --2- 172.21.15.7:0/2625629839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95e0130440 0x7f95e011a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:20.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e58d4700 10 monclient(hunting): get_auth_request con 0x7f95e0130440 auth_method 0 2023-09-30T02:00:20.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e58d4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:00:20.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e58d4700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:00:20.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e58d4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:00:20.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e48d2700 10 monclient(hunting): handle_auth_done global_id 4821 payload 824 2023-09-30T02:00:20.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e48d2700 1 -- 172.21.15.7:0/2625629839 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95e009a900 msgr2=0x7f95e0119de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:20.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e48d2700 1 --2- 172.21.15.7:0/2625629839 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95e009a900 0x7f95e0119de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:20.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e48d2700 1 -- 172.21.15.7:0/2625629839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95e0130440 msgr2=0x7f95e011a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:20.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e48d2700 1 --2- 172.21.15.7:0/2625629839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95e0130440 0x7f95e011a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:20.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e48d2700 10 monclient: _finish_hunting 0 2023-09-30T02:00:20.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e48d2700 1 monclient: found mon.noname-a 2023-09-30T02:00:20.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e48d2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:00:20.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e48d2700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.7:3300/0 2023-09-30T02:00:20.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e48d2700 1 -- 172.21.15.7:0/2625629839 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d0009420 con 0x7f95e009bc40 2023-09-30T02:00:20.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e50d3700 1 --2- 172.21.15.7:0/2625629839 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95e009a900 0x7f95e0119de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:00:20.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.233+0000 7f95e58d4700 1 --2- 172.21.15.7:0/2625629839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95e0130440 0x7f95e011a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:00:20.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.235+0000 7f95e48d2700 1 -- 172.21.15.7:0/2625629839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95e009bc40 msgr2=0x7f95e011a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T02:00:20.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.235+0000 7f95e48d2700 1 --2- 172.21.15.7:0/2625629839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95e009bc40 0x7f95e011a320 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f95d0010db0 tx=0x7f95d00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:20.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.236+0000 7f95d57fa700 1 -- 172.21.15.7:0/2625629839 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d0015070 con 0x7f95e009bc40 2023-09-30T02:00:20.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.236+0000 7f95d57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:00:20.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.236+0000 7f95d57fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:00:20.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.236+0000 7f95d57fa700 10 monclient: dump: 2023-09-30T02:00:20.252 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T02:00:20.252 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T02:00:20.253 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T02:00:20.253 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T02:00:20.253 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T02:00:20.253 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T02:00:20.253 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T02:00:20.253 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T02:00:20.253 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T02:00:20.253 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T02:00:20.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.236+0000 7f95d57fa700 10 monclient: _finish_auth 0 2023-09-30T02:00:20.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.236+0000 7f95d57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:00:20.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.236+0000 7f95e7337700 5 monclient: authenticate success, global_id 4821 2023-09-30T02:00:20.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.236+0000 7f95d57fa700 1 -- 172.21.15.7:0/2625629839 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d000bde0 con 0x7f95e009bc40 2023-09-30T02:00:20.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.236+0000 7f95e7337700 10 monclient: _renew_subs 2023-09-30T02:00:20.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.236+0000 7f95e7337700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T02:00:20.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.236+0000 7f95e7337700 1 -- 172.21.15.7:0/2625629839 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95dc0094f0 con 0x7f95e009bc40 2023-09-30T02:00:20.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.236+0000 7f95d57fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:00:20.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.236+0000 7f95d57fa700 1 -- 172.21.15.7:0/2625629839 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d0019660 con 0x7f95e009bc40 2023-09-30T02:00:20.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.236+0000 7f95d57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:00:20.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.236+0000 7f95d57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:00:20.254 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.236+0000 7f95d57fa700 10 monclient: dump: 2023-09-30T02:00:20.254 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T02:00:20.254 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T02:00:20.255 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T02:00:20.255 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T02:00:20.255 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T02:00:20.255 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T02:00:20.255 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T02:00:20.255 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T02:00:20.255 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T02:00:20.255 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T02:00:20.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.236+0000 7f95d77fe700 4 set_mon_vals no callback set 2023-09-30T02:00:20.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.236+0000 7f95e7337700 10 monclient: _renew_subs 2023-09-30T02:00:20.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.236+0000 7f95e7337700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T02:00:20.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.236+0000 7f95e7337700 1 -- 172.21.15.7:0/2625629839 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95e011b9c0 con 0x7f95e009bc40 2023-09-30T02:00:20.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.237+0000 7f95d57fa700 1 -- 172.21.15.7:0/2625629839 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f95d0019a50 con 0x7f95e009bc40 2023-09-30T02:00:20.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.237+0000 7f95e7337700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:00:20.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.237+0000 7f95e7337700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:00:20.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.237+0000 7f95e7337700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.7:3300/0 2023-09-30T02:00:20.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.237+0000 7f95e7337700 1 -- 172.21.15.7:0/2625629839 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95e00a05d0 con 0x7f95e009bc40 2023-09-30T02:00:20.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.237+0000 7f95d57fa700 1 --2- 172.21.15.7:0/2625629839 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f95c40357e0 0x7f95c4037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:20.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.237+0000 7f95d57fa700 1 -- 172.21.15.7:0/2625629839 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f95d004bd50 con 0x7f95e009bc40 2023-09-30T02:00:20.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.237+0000 7f95e50d3700 1 --2- 172.21.15.7:0/2625629839 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f95c40357e0 0x7f95c4037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:20.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.237+0000 7f95e50d3700 10 monclient: get_auth_request con 0x7f95c40357e0 auth_method 0 2023-09-30T02:00:20.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.240+0000 7f95e50d3700 1 --2- 172.21.15.7:0/2625629839 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f95c40357e0 0x7f95c4037c90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f95dc0030a0 tx=0x7f95dc016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:20.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.242+0000 7f95d57fa700 1 -- 172.21.15.7:0/2625629839 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f95d0019d00 con 0x7f95e009bc40 2023-09-30T02:00:20.256 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.242+0000 7f95d57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:00:20.257 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.242+0000 7f95d57fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:00:20.431 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.431+0000 7f95e7337700 1 -- 172.21.15.7:0/2625629839 --> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f95e0064c10 con 0x7f95c40357e0 2023-09-30T02:00:20.433 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.433+0000 7f95d57fa700 1 -- 172.21.15.7:0/2625629839 <== mgr.4105 v2:172.21.15.50:6824/86200 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53754 (secure 0 0 0) 0x7f95e0064c10 con 0x7f95c40357e0 2023-09-30T02:00:20.434 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-09-30T02:00:20.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.435+0000 7f95e7337700 1 -- 172.21.15.7:0/2625629839 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f95c40357e0 msgr2=0x7f95c4037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:20.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.435+0000 7f95e7337700 1 --2- 172.21.15.7:0/2625629839 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f95c40357e0 0x7f95c4037c90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f95dc0030a0 tx=0x7f95dc016000 comp rx=0 tx=0).stop 2023-09-30T02:00:20.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.435+0000 7f95e7337700 10 monclient: shutdown 2023-09-30T02:00:20.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.435+0000 7f95e7337700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:00:20.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.435+0000 7f95e7337700 1 -- 172.21.15.7:0/2625629839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95e009bc40 msgr2=0x7f95e011a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:20.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.435+0000 7f95e7337700 1 --2- 172.21.15.7:0/2625629839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95e009bc40 0x7f95e011a320 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f95d0010db0 tx=0x7f95d00091a0 comp rx=0 tx=0).stop 2023-09-30T02:00:20.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.435+0000 7f95e7337700 1 -- 172.21.15.7:0/2625629839 shutdown_connections 2023-09-30T02:00:20.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.435+0000 7f95e7337700 1 --2- 172.21.15.7:0/2625629839 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f95c40357e0 0x7f95c4037c90 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:20.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.435+0000 7f95e7337700 1 --2- 172.21.15.7:0/2625629839 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95e0130440 0x7f95e011a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:20.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.435+0000 7f95e7337700 1 --2- 172.21.15.7:0/2625629839 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f95e009a900 0x7f95e0119de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:20.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.435+0000 7f95e7337700 1 --2- 172.21.15.7:0/2625629839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f95e009bc40 0x7f95e011a320 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:20.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.435+0000 7f95e7337700 1 -- 172.21.15.7:0/2625629839 >> 172.21.15.7:0/2625629839 conn(0x7f95e009e0b0 msgr2=0x7f95e006f610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:20.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.435+0000 7f95e7337700 1 -- 172.21.15.7:0/2625629839 shutdown_connections 2023-09-30T02:00:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.435+0000 7f95e7337700 1 -- 172.21.15.7:0/2625629839 wait complete. 2023-09-30T02:00:20.437 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-09-30T02:00:20.447 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2023-09-30T02:00:20.334243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":32644,"kb_used_data":1876,"kb_used_omap":0,"kb_used_meta":30720,"kb_avail":566198396,"statfs":{"total":579820584960,"available":579787157504,"internally_reserved":0,"allocated":1921024,"data_stored":1316285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31457280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002491"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T02:00:08.487442+0000","last_change":"2023-09-30T02:00:08.487410+0000","last_active":"2023-09-30T02:00:08.487442+0000","last_peered":"2023-09-30T02:00:08.487442+0000","last_clean":"2023-09-30T02:00:08.487442+0000","last_became_active":"2023-09-30T01:54:47.042183+0000","last_became_peered":"2023-09-30T01:54:47.042183+0000","last_unstale":"2023-09-30T02:00:08.487442+0000","last_undegraded":"2023-09-30T02:00:08.487442+0000","last_fullsized":"2023-09-30T02:00:08.487442+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:00:08.487298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:00:08.487298+0000","last_clean_scrub_stamp":"2023-09-30T02:00:08.487298+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:06:54.316214+0000","scrub_duration":0.0015499820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T02:00:02.125333+0000","last_change":"2023-09-30T02:00:02.125304+0000","last_active":"2023-09-30T02:00:02.125333+0000","last_peered":"2023-09-30T02:00:02.125333+0000","last_clean":"2023-09-30T02:00:02.125333+0000","last_became_active":"2023-09-30T01:54:46.896620+0000","last_became_peered":"2023-09-30T01:54:46.896620+0000","last_unstale":"2023-09-30T02:00:02.125333+0000","last_undegraded":"2023-09-30T02:00:02.125333+0000","last_fullsized":"2023-09-30T02:00:02.125333+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:00:02.125199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:00:02.125199+0000","last_clean_scrub_stamp":"2023-09-30T02:00:02.125199+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:02:27.403478+0000","scrub_duration":0.001956008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T02:00:05.495146+0000","last_change":"2023-09-30T02:00:05.495118+0000","last_active":"2023-09-30T02:00:05.495146+0000","last_peered":"2023-09-30T02:00:05.495146+0000","last_clean":"2023-09-30T02:00:05.495146+0000","last_became_active":"2023-09-30T01:54:47.041671+0000","last_became_peered":"2023-09-30T01:54:47.041671+0000","last_unstale":"2023-09-30T02:00:05.495146+0000","last_undegraded":"2023-09-30T02:00:05.495146+0000","last_fullsized":"2023-09-30T02:00:05.495146+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:00:05.494986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:00:05.494986+0000","last_clean_scrub_stamp":"2023-09-30T02:00:05.494986+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:03:07.763397+0000","scrub_duration":0.001875968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T02:00:06.496536+0000","last_change":"2023-09-30T02:00:06.496505+0000","last_active":"2023-09-30T02:00:06.496536+0000","last_peered":"2023-09-30T02:00:06.496536+0000","last_clean":"2023-09-30T02:00:06.496536+0000","last_became_active":"2023-09-30T01:54:46.896191+0000","last_became_peered":"2023-09-30T01:54:46.896191+0000","last_unstale":"2023-09-30T02:00:06.496536+0000","last_undegraded":"2023-09-30T02:00:06.496536+0000","last_fullsized":"2023-09-30T02:00:06.496536+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:00:06.496406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:00:06.496406+0000","last_clean_scrub_stamp":"2023-09-30T02:00:06.496406+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:37:01.810576+0000","scrub_duration":0.0016345349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'119","reported_seq":173,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T02:00:00.449572+0000","last_change":"2023-09-30T02:00:00.449540+0000","last_active":"2023-09-30T02:00:00.449572+0000","last_peered":"2023-09-30T02:00:00.449572+0000","last_clean":"2023-09-30T02:00:00.449572+0000","last_became_active":"2023-09-30T01:54:45.896315+0000","last_became_peered":"2023-09-30T01:54:45.896315+0000","last_unstale":"2023-09-30T02:00:00.449572+0000","last_undegraded":"2023-09-30T02:00:00.449572+0000","last_fullsized":"2023-09-30T02:00:00.449572+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'119","last_scrub_stamp":"2023-09-30T02:00:00.449441+0000","last_deep_scrub":"15'119","last_deep_scrub_stamp":"2023-09-30T02:00:00.449441+0000","last_clean_scrub_stamp":"2023-09-30T02:00:00.449441+0000","objects_scrubbed":2,"log_size":119,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:19:19.566659+0000","scrub_duration":0.0042002899999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T02:00:06.679899+0000","last_change":"2023-09-30T02:00:06.679869+0000","last_active":"2023-09-30T02:00:06.679899+0000","last_peered":"2023-09-30T02:00:06.679899+0000","last_clean":"2023-09-30T02:00:06.679899+0000","last_became_active":"2023-09-30T01:54:46.898402+0000","last_became_peered":"2023-09-30T01:54:46.898402+0000","last_unstale":"2023-09-30T02:00:06.679899+0000","last_undegraded":"2023-09-30T02:00:06.679899+0000","last_fullsized":"2023-09-30T02:00:06.679899+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:00:06.679744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:00:06.679744+0000","last_clean_scrub_stamp":"2023-09-30T02:00:06.679744+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:21:39.468872+0000","scrub_duration":0.001931014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T01:59:58.327166+0000","last_change":"2023-09-30T01:59:58.327146+0000","last_active":"2023-09-30T01:59:58.327166+0000","last_peered":"2023-09-30T01:59:58.327166+0000","last_clean":"2023-09-30T01:59:58.327166+0000","last_became_active":"2023-09-30T01:54:46.895973+0000","last_became_peered":"2023-09-30T01:54:46.895973+0000","last_unstale":"2023-09-30T01:59:58.327166+0000","last_undegraded":"2023-09-30T01:59:58.327166+0000","last_fullsized":"2023-09-30T01:59:58.327166+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T01:59:58.327065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T01:59:58.327065+0000","last_clean_scrub_stamp":"2023-09-30T01:59:58.327065+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:09:20.050882+0000","scrub_duration":0.0013473580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T02:00:03.750114+0000","last_change":"2023-09-30T02:00:03.750085+0000","last_active":"2023-09-30T02:00:03.750114+0000","last_peered":"2023-09-30T02:00:03.750114+0000","last_clean":"2023-09-30T02:00:03.750114+0000","last_became_active":"2023-09-30T01:54:46.898101+0000","last_became_peered":"2023-09-30T01:54:46.898101+0000","last_unstale":"2023-09-30T02:00:03.750114+0000","last_undegraded":"2023-09-30T02:00:03.750114+0000","last_fullsized":"2023-09-30T02:00:03.750114+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:00:03.749972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:00:03.749972+0000","last_clean_scrub_stamp":"2023-09-30T02:00:03.749972+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:45:43.862699+0000","scrub_duration":0.002195093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T02:00:06.175809+0000","last_change":"2023-09-30T02:00:06.175777+0000","last_active":"2023-09-30T02:00:06.175809+0000","last_peered":"2023-09-30T02:00:06.175809+0000","last_clean":"2023-09-30T02:00:06.175809+0000","last_became_active":"2023-09-30T01:54:47.041914+0000","last_became_peered":"2023-09-30T01:54:47.041914+0000","last_unstale":"2023-09-30T02:00:06.175809+0000","last_undegraded":"2023-09-30T02:00:06.175809+0000","last_fullsized":"2023-09-30T02:00:06.175809+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:00:06.175667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:00:06.175667+0000","last_clean_scrub_stamp":"2023-09-30T02:00:06.175667+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:10:20.771452+0000","scrub_duration":0.0016157400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640327,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5076,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366764,"statfs":{"total":96636764160,"available":96631566336,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52500000000000002,"15min":0.54300000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.69199999999999995,"5min":0.747,"15min":0.747},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53500000000000003,"15min":0.56100000000000005},"min":{"1min":0.40799999999999997,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66400000000000003,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55200000000000005}]},{"osd":1,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.51500000000000001,"15min":0.53700000000000003},"min":{"1min":0.376,"5min":0.313,"15min":0.313},"max":{"1min":0.66400000000000003,"5min":0.751,"15min":0.751},"last":0.503},{"interface":"front","average":{"1min":0.498,"5min":0.53500000000000003,"15min":0.56100000000000005},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.629,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.56699999999999995}]},{"osd":2,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55100000000000005,"15min":0.57299999999999995},"min":{"1min":0.42499999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.73899999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55300000000000005,"15min":0.56499999999999995},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72599999999999998,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.49399999999999999}]},{"osd":3,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.41999999999999998,"15min":0.443},"min":{"1min":0.29899999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.60499999999999998,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45500000000000002,"15min":0.47899999999999998},"min":{"1min":0.312,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.38400000000000001}]},{"osd":4,"last update":"Sat Sep 30 01:59:48 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.45500000000000002,"15min":0.48999999999999999},"min":{"1min":0.311,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.50900000000000001,"15min":0.53500000000000003},"min":{"1min":0.36599999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.76700000000000002,"5min":0.874,"15min":0.874},"last":0.46000000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640325,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5076,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366764,"statfs":{"total":96636764160,"available":96631566336,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.51000000000000001,"15min":0.51400000000000001},"min":{"1min":0.35399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70199999999999996,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.51600000000000001,"15min":0.48799999999999999},"min":{"1min":0.38800000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.68799999999999994,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.54600000000000004}]},{"osd":1,"last update":"Sat Sep 30 01:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51700000000000002,"15min":0.49099999999999999},"min":{"1min":0.39500000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.65100000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53500000000000003,"15min":0.52500000000000002},"min":{"1min":0.40200000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.65600000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.58299999999999996}]},{"osd":2,"last update":"Sat Sep 30 01:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.50600000000000001},"min":{"1min":0.32900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.72799999999999998,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.435},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55200000000000005,"15min":0.52600000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.76100000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.60999999999999999}]},{"osd":3,"last update":"Sat Sep 30 01:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40799999999999997,"15min":0.40999999999999998},"min":{"1min":0.28299999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.432,"5min":0.44800000000000001,"15min":0.45500000000000002},"min":{"1min":0.33200000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.624,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.503}]},{"osd":5,"last update":"Sat Sep 30 01:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.45100000000000001,"15min":0.45500000000000002},"min":{"1min":0.31,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.67100000000000004,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.48699999999999999,"15min":0.501},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.68899999999999995,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.48699999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640325,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6108,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365732,"statfs":{"total":96636764160,"available":96630509568,"internally_reserved":0,"allocated":675840,"data_stored":572143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:59:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.54400000000000004,"15min":0.55200000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.5},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54400000000000004,"15min":0.54300000000000004},"min":{"1min":0.39600000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.746,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.48399999999999999}]},{"osd":1,"last update":"Sat Sep 30 01:59:55 2023","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51200000000000001,"15min":0.51100000000000001},"min":{"1min":0.38400000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.622,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.50700000000000001,"15min":0.497},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70999999999999996,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.42299999999999999}]},{"osd":2,"last update":"Sat Sep 30 01:59:55 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53700000000000003,"15min":0.54700000000000004},"min":{"1min":0.34100000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69199999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.46300000000000002}]},{"osd":4,"last update":"Sat Sep 30 01:59:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.39700000000000002,"15min":0.39900000000000002},"min":{"1min":0.29399999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.60599999999999998,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.309},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.47799999999999998,"15min":0.498},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.66500000000000004,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.56100000000000005}]},{"osd":5,"last update":"Sat Sep 30 01:59:55 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.44900000000000001,"15min":0.47099999999999997},"min":{"1min":0.27600000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.67300000000000004,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46400000000000002,"15min":0.47999999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.75700000000000001,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.52200000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640326,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5140,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366700,"statfs":{"total":96636764160,"available":96631500800,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.35599999999999998,"15min":0.35299999999999998},"min":{"1min":0.24099999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.55000000000000004,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.39300000000000002,"15min":0.38500000000000001},"min":{"1min":0.313,"5min":0.254,"15min":0.254},"max":{"1min":0.56599999999999995,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.34999999999999998}]},{"osd":1,"last update":"Sat Sep 30 01:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.36199999999999999,"15min":0.36799999999999999},"min":{"1min":0.25900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.51700000000000002,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.375,"15min":0.36299999999999999},"min":{"1min":0.245,"5min":0.222,"15min":0.222},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.308}]},{"osd":3,"last update":"Sat Sep 30 01:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71999999999999997,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57199999999999995,"15min":0.56399999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.755,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.63}]},{"osd":4,"last update":"Sat Sep 30 01:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.58099999999999996,"15min":0.57699999999999996},"min":{"1min":0.41899999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70899999999999996,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.58999999999999997,"15min":0.58199999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77600000000000002,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.60999999999999999}]},{"osd":5,"last update":"Sat Sep 30 01:59:57 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.57799999999999996,"15min":0.56899999999999995},"min":{"1min":0.40899999999999997,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73699999999999999,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.60499999999999998,"15min":0.60899999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.73099999999999998,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.57699999999999996}]}]},{"osd":0,"up_from":11,"seq":47244640325,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6104,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94365736,"statfs":{"total":96636764160,"available":96630513664,"internally_reserved":0,"allocated":671744,"data_stored":571282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 01:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.34100000000000003,"15min":0.33400000000000002},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.48799999999999999,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.376},{"interface":"front","average":{"1min":0.377,"5min":0.39100000000000001,"15min":0.39600000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.51000000000000001,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.40500000000000003}]},{"osd":2,"last update":"Sat Sep 30 01:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.35699999999999998,"15min":0.35999999999999999},"min":{"1min":0.21299999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.53200000000000003,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.41599999999999998,"15min":0.42599999999999999},"min":{"1min":0.28299999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.54000000000000004,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.47299999999999998}]},{"osd":3,"last update":"Sat Sep 30 01:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39100000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.75,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.55800000000000005,"15min":0.56000000000000005},"min":{"1min":0.39900000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.76900000000000002,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.49299999999999999}]},{"osd":4,"last update":"Sat Sep 30 01:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57599999999999996,"15min":0.57899999999999996},"min":{"1min":0.33400000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.72199999999999998,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56899999999999995,"15min":0.57599999999999996},"min":{"1min":0.34899999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73499999999999999,"5min":1.026,"15min":1.026},"last":0.57399999999999995}]},{"osd":5,"last update":"Sat Sep 30 01:59:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56599999999999995,"15min":0.57599999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76600000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.58699999999999997,"15min":0.58899999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.71199999999999997,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.58799999999999997}]}]},{"osd":1,"up_from":11,"seq":47244640327,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5140,"kb_used_data":140,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366700,"statfs":{"total":96636764160,"available":96631500800,"internally_reserved":0,"allocated":143360,"data_stored":43215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 01:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.34000000000000002,"15min":0.312},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.52900000000000003,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.34699999999999998,"15min":0.32200000000000001},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.50900000000000001,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.38700000000000001}]},{"osd":2,"last update":"Sat Sep 30 01:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.36099999999999999,"15min":0.33800000000000002},"min":{"1min":0.224,"5min":0.22,"15min":0.22},"max":{"1min":0.57799999999999996,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.375,"15min":0.36399999999999999},"min":{"1min":0.27100000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.55600000000000005,"5min":0.62,"15min":0.62},"last":0.36699999999999999}]},{"osd":3,"last update":"Sat Sep 30 01:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.54400000000000004,"15min":0.52000000000000002},"min":{"1min":0.46000000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.496},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.373,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73299999999999998,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46000000000000002}]},{"osd":4,"last update":"Sat Sep 30 01:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57599999999999996,"15min":0.56399999999999995},"min":{"1min":0.40000000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.751,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.441,"5min":0.371,"15min":0.371},"max":{"1min":0.72899999999999998,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.51700000000000002}]},{"osd":5,"last update":"Sat Sep 30 01:59:52 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57299999999999995,"15min":0.55600000000000005},"min":{"1min":0.42799999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59299999999999997,"15min":0.57199999999999995},"min":{"1min":0.47599999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80800000000000005,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.56299999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:00:20.450 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-09-30T02:00:20.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.568+0000 7f70f6f88700 10 monclient: get_monmap_and_config 2023-09-30T02:00:20.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.568+0000 7f70f6f88700 10 monclient: build_initial_monmap 2023-09-30T02:00:20.568 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.568+0000 7f70f6f88700 10 monclient: monmap: 2023-09-30T02:00:20.568 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T02:00:20.568 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T02:00:20.568 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T02:00:20.569368+0000 2023-09-30T02:00:20.569 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T02:00:20.569368+0000 2023-09-30T02:00:20.569 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T02:00:20.569 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T02:00:20.569 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a-legacy 2023-09-30T02:00:20.569 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.50:6789/0 mon.noname-c-legacy 2023-09-30T02:00:20.569 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-b 2023-09-30T02:00:20.569 INFO:teuthology.orchestra.run.smithi007.stderr:3: v2:172.21.15.7:3300/0 mon.noname-a 2023-09-30T02:00:20.569 INFO:teuthology.orchestra.run.smithi007.stderr:4: v2:172.21.15.50:3300/0 mon.noname-c 2023-09-30T02:00:20.569 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T02:00:20.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.568+0000 7f70f6f88700 1 Processor -- start 2023-09-30T02:00:20.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f6f88700 1 -- start start 2023-09-30T02:00:20.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f6f88700 10 monclient: init 2023-09-30T02:00:20.579 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f6f88700 10 monclient: _reopen_session rank -1 2023-09-30T02:00:20.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f6f88700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-09-30T02:00:20.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f6f88700 1 --2- >> v2:172.21.15.50:3300/0 conn(0x7f70f00a29e0 0x7f70f0096a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:20.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f6f88700 10 monclient(hunting): picked mon.noname-c con 0x7f70f00a29e0 addr v2:172.21.15.50:3300/0 2023-09-30T02:00:20.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f6f88700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f70f0096f90 addr v1:172.21.15.50:6789/0 2023-09-30T02:00:20.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f6f88700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f70f0097390 0x7f70f0132930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:20.580 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f6f88700 10 monclient(hunting): picked mon.noname-b con 0x7f70f0097390 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T02:00:20.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f6f88700 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f0069cc0 con 0x7f70f0096f90 2023-09-30T02:00:20.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f6f88700 10 monclient(hunting): start opening mon connection 2023-09-30T02:00:20.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f6f88700 1 -- --> v2:172.21.15.50:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70f0098ea0 con 0x7f70f00a29e0 2023-09-30T02:00:20.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f6f88700 10 monclient(hunting): start opening mon connection 2023-09-30T02:00:20.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f6f88700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f00a2db0 con 0x7f70f0097390 2023-09-30T02:00:20.581 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f6f88700 10 monclient(hunting): _renew_subs 2023-09-30T02:00:20.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f6f88700 10 monclient(hunting): authenticate will time out at 1509.634543s 2023-09-30T02:00:20.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f5525700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f70f0097390 0x7f70f0132930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:20.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f5525700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f70f0097390 0x7f70f0132930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:47862/0 (socket says 172.21.15.7:47862) 2023-09-30T02:00:20.582 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f5525700 1 -- 172.21.15.7:0/3205959726 learned_addr learned my addr 172.21.15.7:0/3205959726 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T02:00:20.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f5525700 10 monclient(hunting): get_auth_request con 0x7f70f0097390 auth_method 0 2023-09-30T02:00:20.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f5525700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:00:20.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f5525700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:00:20.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f5525700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:00:20.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f5525700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:00:20.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f5525700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:00:20.583 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f5525700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:00:20.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f4d24700 1 --2- 172.21.15.7:0/3205959726 >> v2:172.21.15.50:3300/0 conn(0x7f70f00a29e0 0x7f70f0096a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:20.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f4d24700 10 monclient(hunting): get_auth_request con 0x7f70f00a29e0 auth_method 0 2023-09-30T02:00:20.584 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f4d24700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:00:20.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f4d24700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:00:20.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f4d24700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:00:20.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f5525700 10 monclient(hunting): handle_auth_done global_id 4850 payload 274 2023-09-30T02:00:20.585 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f5525700 1 -- 172.21.15.7:0/3205959726 >> v2:172.21.15.50:3300/0 conn(0x7f70f00a29e0 msgr2=0x7f70f0096a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:20.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f5525700 1 --2- 172.21.15.7:0/3205959726 >> v2:172.21.15.50:3300/0 conn(0x7f70f00a29e0 0x7f70f0096a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:20.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f5525700 1 -- 172.21.15.7:0/3205959726 >> v1:172.21.15.50:6789/0 conn(0x7f70f0096f90 legacy=0x7f70f0132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:00:20.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f5525700 10 monclient: _finish_hunting 0 2023-09-30T02:00:20.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f5525700 1 monclient: found mon.noname-b 2023-09-30T02:00:20.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f5525700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:00:20.586 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f5525700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.7:3301/0 2023-09-30T02:00:20.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.569+0000 7f70f5525700 1 -- 172.21.15.7:0/3205959726 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f0098270 con 0x7f70f0097390 2023-09-30T02:00:20.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70f5525700 1 -- 172.21.15.7:0/3205959726 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f70f0097390 msgr2=0x7f70f0132930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T02:00:20.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70f5525700 1 --2- 172.21.15.7:0/3205959726 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f70f0097390 0x7f70f0132930 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f70e8009f20 tx=0x7f70e8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f4af610c7972bd9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:20.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70ef7fe700 1 -- 172.21.15.7:0/3205959726 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e800e070 con 0x7f70f0097390 2023-09-30T02:00:20.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70ef7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:00:20.587 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70ef7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:00:20.588 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70ef7fe700 10 monclient: dump: 2023-09-30T02:00:20.588 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T02:00:20.588 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T02:00:20.588 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T02:00:20.588 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T02:00:20.588 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T02:00:20.589 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T02:00:20.589 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T02:00:20.589 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T02:00:20.589 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T02:00:20.589 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T02:00:20.589 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70ef7fe700 10 monclient: _finish_auth 0 2023-09-30T02:00:20.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70ef7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:00:20.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70f6f88700 5 monclient: authenticate success, global_id 4850 2023-09-30T02:00:20.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70f6f88700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:00:20.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70ef7fe700 1 -- 172.21.15.7:0/3205959726 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e800bc50 con 0x7f70f0097390 2023-09-30T02:00:20.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70ef7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:00:20.590 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70f6f88700 10 monclient: get_monmap_and_config success 2023-09-30T02:00:20.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70f6f88700 4 set_mon_vals no callback set 2023-09-30T02:00:20.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70ef7fe700 1 -- 172.21.15.7:0/3205959726 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e80092e0 con 0x7f70f0097390 2023-09-30T02:00:20.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70ef7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:00:20.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70ef7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:00:20.591 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70ef7fe700 10 monclient: dump: 2023-09-30T02:00:20.591 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T02:00:20.592 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T02:00:20.592 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T02:00:20.592 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T02:00:20.592 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T02:00:20.592 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T02:00:20.592 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T02:00:20.593 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T02:00:20.593 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T02:00:20.593 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T02:00:20.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70f6f88700 10 monclient: shutdown 2023-09-30T02:00:20.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70f6f88700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:00:20.593 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70f6f88700 1 -- 172.21.15.7:0/3205959726 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f70f0097390 msgr2=0x7f70f0132930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:20.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70f6f88700 1 --2- 172.21.15.7:0/3205959726 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f70f0097390 0x7f70f0132930 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f70e8009f20 tx=0x7f70e8009840 comp rx=0 tx=0).stop 2023-09-30T02:00:20.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70f6f88700 1 -- 172.21.15.7:0/3205959726 shutdown_connections 2023-09-30T02:00:20.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70f6f88700 1 --2- 172.21.15.7:0/3205959726 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f70f0097390 0x7f70f0132930 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:20.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70f6f88700 1 --2- 172.21.15.7:0/3205959726 >> v2:172.21.15.50:3300/0 conn(0x7f70f00a29e0 0x7f70f0096a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:20.594 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70f6f88700 1 -- 172.21.15.7:0/3205959726 >> 172.21.15.7:0/3205959726 conn(0x7f70f00710d0 msgr2=0x7f70f00714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:20.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70f6f88700 1 -- 172.21.15.7:0/3205959726 shutdown_connections 2023-09-30T02:00:20.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.572+0000 7f70f6f88700 1 -- 172.21.15.7:0/3205959726 wait complete. 2023-09-30T02:00:20.595 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.573+0000 7f70f6f88700 10 monclient: build_initial_monmap 2023-09-30T02:00:20.596 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.573+0000 7f70f6f88700 10 monclient: monmap: 2023-09-30T02:00:20.596 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2023-09-30T02:00:20.596 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:00:20.596 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2023-09-30T02:00:20.596 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2023-09-30T02:00:20.597 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2023-09-30T02:00:20.597 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T02:00:20.597 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.noname-a 2023-09-30T02:00:20.597 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.noname-b 2023-09-30T02:00:20.597 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.noname-c 2023-09-30T02:00:20.597 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T02:00:20.597 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.573+0000 7f70f6f88700 1 Processor -- start 2023-09-30T02:00:20.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.573+0000 7f70f6f88700 1 -- start start 2023-09-30T02:00:20.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.573+0000 7f70f6f88700 10 monclient: init 2023-09-30T02:00:20.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f6f88700 10 monclient: _reopen_session rank -1 2023-09-30T02:00:20.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f6f88700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:00:20.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f6f88700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f70f0096f90 0x7f70f0141d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:20.598 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f6f88700 10 monclient(hunting): picked mon.noname-c con 0x7f70f0096f90 addr [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-09-30T02:00:20.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f6f88700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70f0097390 0x7f70f009dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:20.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f6f88700 10 monclient(hunting): picked mon.noname-a con 0x7f70f0097390 addr [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] 2023-09-30T02:00:20.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f6f88700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70f00a29e0 0x7f70f009e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:20.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f6f88700 10 monclient(hunting): picked mon.noname-b con 0x7f70f00a29e0 addr [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2023-09-30T02:00:20.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f6f88700 10 monclient(hunting): start opening mon connection 2023-09-30T02:00:20.599 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f6f88700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f00a1220 con 0x7f70f0097390 2023-09-30T02:00:20.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f6f88700 10 monclient(hunting): start opening mon connection 2023-09-30T02:00:20.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f6f88700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f01350b0 con 0x7f70f00a29e0 2023-09-30T02:00:20.600 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f6f88700 10 monclient(hunting): start opening mon connection 2023-09-30T02:00:20.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f6f88700 1 -- --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f009fec0 con 0x7f70f0096f90 2023-09-30T02:00:20.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f6f88700 10 monclient(hunting): _renew_subs 2023-09-30T02:00:20.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f6f88700 10 monclient(hunting): authenticate will time out at 1509.639443s 2023-09-30T02:00:20.601 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f4d24700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f70f0096f90 0x7f70f0141d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:20.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f4d24700 1 --2- >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f70f0096f90 0x7f70f0141d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3301/0 says I am v2:172.21.15.7:47872/0 (socket says 172.21.15.7:47872) 2023-09-30T02:00:20.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f4d24700 1 -- 172.21.15.7:0/804126565 learned_addr learned my addr 172.21.15.7:0/804126565 (peer_addr_for_me v2:172.21.15.7:0/0) 2023-09-30T02:00:20.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f4d24700 10 monclient(hunting): get_auth_request con 0x7f70f0096f90 auth_method 0 2023-09-30T02:00:20.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f4d24700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:00:20.602 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f4d24700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:00:20.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f4d24700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:00:20.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70effff700 1 --2- 172.21.15.7:0/804126565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70f0097390 0x7f70f009dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:20.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70effff700 10 monclient(hunting): get_auth_request con 0x7f70f0097390 auth_method 0 2023-09-30T02:00:20.603 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70effff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:00:20.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70effff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:00:20.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70effff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:00:20.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f4d24700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:00:20.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f4d24700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:00:20.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f4d24700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:00:20.604 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70effff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:00:20.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70effff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:00:20.605 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70effff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:00:20.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f4d24700 10 monclient(hunting): handle_auth_done global_id 4853 payload 824 2023-09-30T02:00:20.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f5525700 1 --2- 172.21.15.7:0/804126565 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70f00a29e0 0x7f70f009e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:20.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f4d24700 1 -- 172.21.15.7:0/804126565 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70f00a29e0 msgr2=0x7f70f009e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:20.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f4d24700 1 --2- 172.21.15.7:0/804126565 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70f00a29e0 0x7f70f009e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:20.606 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f4d24700 1 -- 172.21.15.7:0/804126565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70f0097390 msgr2=0x7f70f009dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:20.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f4d24700 1 --2- 172.21.15.7:0/804126565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70f0097390 0x7f70f009dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:20.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f4d24700 10 monclient: _finish_hunting 0 2023-09-30T02:00:20.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f4d24700 1 monclient: found mon.noname-c 2023-09-30T02:00:20.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f4d24700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:00:20.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f4d24700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.7:3301/0 2023-09-30T02:00:20.607 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70f4d24700 1 -- 172.21.15.7:0/804126565 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e8009ae0 con 0x7f70f0096f90 2023-09-30T02:00:20.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.574+0000 7f70effff700 1 --2- 172.21.15.7:0/804126565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70f0097390 0x7f70f009dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:00:20.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70f4d24700 1 -- 172.21.15.7:0/804126565 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f70f0096f90 msgr2=0x7f70f0141d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T02:00:20.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70f4d24700 1 --2- 172.21.15.7:0/804126565 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f70f0096f90 0x7f70f0141d10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f70e400d810 tx=0x7f70e4008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:20.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70ecff9700 1 -- 172.21.15.7:0/804126565 <== mon.2 v2:172.21.15.7:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e4012800 con 0x7f70f0096f90 2023-09-30T02:00:20.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70ecff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:00:20.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70ecff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:00:20.609 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70ecff9700 10 monclient: dump: 2023-09-30T02:00:20.609 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T02:00:20.609 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T02:00:20.609 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T02:00:20.609 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T02:00:20.610 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T02:00:20.610 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T02:00:20.610 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T02:00:20.610 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T02:00:20.610 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T02:00:20.610 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T02:00:20.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70ecff9700 10 monclient: _finish_auth 0 2023-09-30T02:00:20.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:00:20.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70ecff9700 1 -- 172.21.15.7:0/804126565 <== mon.2 v2:172.21.15.7:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e4012960 con 0x7f70f0096f90 2023-09-30T02:00:20.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70f6f88700 5 monclient: authenticate success, global_id 4853 2023-09-30T02:00:20.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70f6f88700 10 monclient: _renew_subs 2023-09-30T02:00:20.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70f6f88700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T02:00:20.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70f6f88700 1 -- 172.21.15.7:0/804126565 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70f011c730 con 0x7f70f0096f90 2023-09-30T02:00:20.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70ecff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:00:20.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70ecff9700 1 -- 172.21.15.7:0/804126565 <== mon.2 v2:172.21.15.7:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e4012d00 con 0x7f70f0096f90 2023-09-30T02:00:20.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70ecff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:00:20.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70ecff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:00:20.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70eeffd700 4 set_mon_vals no callback set 2023-09-30T02:00:20.611 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70ecff9700 10 monclient: dump: 2023-09-30T02:00:20.611 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2023-09-30T02:00:20.612 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 2ea1044d-d984-49b9-baad-7f36bd454a19 2023-09-30T02:00:20.612 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2023-09-30T01:54:12.237593+0000 2023-09-30T02:00:20.612 INFO:teuthology.orchestra.run.smithi007.stderr:created 2023-09-30T01:54:12.237593+0000 2023-09-30T02:00:20.612 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 17 (quincy) 2023-09-30T02:00:20.612 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2023-09-30T02:00:20.612 INFO:teuthology.orchestra.run.smithi007.stderr:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-09-30T02:00:20.612 INFO:teuthology.orchestra.run.smithi007.stderr:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2023-09-30T02:00:20.612 INFO:teuthology.orchestra.run.smithi007.stderr:2: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2023-09-30T02:00:20.612 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-09-30T02:00:20.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70f6f88700 10 monclient: _renew_subs 2023-09-30T02:00:20.612 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70f6f88700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T02:00:20.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.577+0000 7f70f6f88700 1 -- 172.21.15.7:0/804126565 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f009f940 con 0x7f70f0096f90 2023-09-30T02:00:20.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.578+0000 7f70ecff9700 1 -- 172.21.15.7:0/804126565 <== mon.2 v2:172.21.15.7:3301/0 4 ==== mgrmap(e 5) v1 ==== 42541+0+0 (secure 0 0 0) 0x7f70e4014640 con 0x7f70f0096f90 2023-09-30T02:00:20.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.578+0000 7f70f6f88700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:00:20.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.578+0000 7f70f6f88700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:00:20.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.578+0000 7f70f6f88700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T02:00:20.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.578+0000 7f70f6f88700 1 -- 172.21.15.7:0/804126565 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70f0064c10 con 0x7f70f0096f90 2023-09-30T02:00:20.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.578+0000 7f70ecff9700 1 --2- 172.21.15.7:0/804126565 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f70d40357e0 0x7f70d4037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:00:20.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.578+0000 7f70ecff9700 1 -- 172.21.15.7:0/804126565 <== mon.2 v2:172.21.15.7:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4342+0+0 (secure 0 0 0) 0x7f70e4048380 con 0x7f70f0096f90 2023-09-30T02:00:20.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.578+0000 7f70effff700 1 --2- 172.21.15.7:0/804126565 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f70d40357e0 0x7f70d4037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:00:20.613 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.578+0000 7f70effff700 10 monclient: get_auth_request con 0x7f70d40357e0 auth_method 0 2023-09-30T02:00:20.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.581+0000 7f70effff700 1 --2- 172.21.15.7:0/804126565 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f70d40357e0 0x7f70d4037c90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f70f009f540 tx=0x7f70dc007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:00:20.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.582+0000 7f70ecff9700 1 -- 172.21.15.7:0/804126565 <== mon.2 v2:172.21.15.7:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f70e4012e60 con 0x7f70f0096f90 2023-09-30T02:00:20.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.582+0000 7f70ecff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:00:20.614 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.582+0000 7f70ecff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:00:20.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.775+0000 7f70f6f88700 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-09-30T02:00:20.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.775+0000 7f70f6f88700 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-09-30T02:00:20.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.775+0000 7f70f6f88700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.7:3301/0 2023-09-30T02:00:20.775 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.775+0000 7f70f6f88700 1 -- 172.21.15.7:0/804126565 --> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f70f0099510 con 0x7f70f0096f90 2023-09-30T02:00:20.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.778+0000 7f70ecff9700 1 -- 172.21.15.7:0/804126565 <== mon.2 v2:172.21.15.7:3301/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f70e4007450 con 0x7f70f0096f90 2023-09-30T02:00:20.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.778+0000 7f70ecff9700 10 monclient: handle_config config(1 keys) v1 2023-09-30T02:00:20.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.778+0000 7f70eeffd700 4 set_mon_vals no callback set 2023-09-30T02:00:20.778 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.778+0000 7f70eeffd700 10 set_mon_vals mon_health_to_clog = false 2023-09-30T02:00:20.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.778+0000 7f70ecff9700 1 -- 172.21.15.7:0/804126565 <== mon.2 v2:172.21.15.7:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7f70e4011340 con 0x7f70f0096f90 2023-09-30T02:00:20.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.778+0000 7f70ecff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-09-30T02:00:20.779 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.778+0000 7f70ecff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:00:20.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.781+0000 7f70f6f88700 1 -- 172.21.15.7:0/804126565 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f70d40357e0 msgr2=0x7f70d4037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:20.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.781+0000 7f70f6f88700 1 --2- 172.21.15.7:0/804126565 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f70d40357e0 0x7f70d4037c90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f70f009f540 tx=0x7f70dc007ea0 comp rx=0 tx=0).stop 2023-09-30T02:00:20.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.781+0000 7f70f6f88700 10 monclient: shutdown 2023-09-30T02:00:20.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.781+0000 7f70f6f88700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:00:20.781 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.781+0000 7f70f6f88700 1 -- 172.21.15.7:0/804126565 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f70f0096f90 msgr2=0x7f70f0141d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:00:20.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.781+0000 7f70f6f88700 1 --2- 172.21.15.7:0/804126565 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f70f0096f90 0x7f70f0141d10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f70e400d810 tx=0x7f70e4008040 comp rx=0 tx=0).stop 2023-09-30T02:00:20.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.781+0000 7f70f6f88700 1 -- 172.21.15.7:0/804126565 shutdown_connections 2023-09-30T02:00:20.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.781+0000 7f70f6f88700 1 --2- 172.21.15.7:0/804126565 >> [v2:172.21.15.50:6824/86200,v1:172.21.15.50:6825/86200] conn(0x7f70d40357e0 0x7f70d4037c90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:20.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.781+0000 7f70f6f88700 1 --2- 172.21.15.7:0/804126565 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70f00a29e0 0x7f70f009e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:20.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.781+0000 7f70f6f88700 1 --2- 172.21.15.7:0/804126565 >> [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] conn(0x7f70f0096f90 0x7f70f0141d10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:20.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.781+0000 7f70f6f88700 1 --2- 172.21.15.7:0/804126565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f70f0097390 0x7f70f009dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:00:20.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.781+0000 7f70f6f88700 1 -- 172.21.15.7:0/804126565 >> 172.21.15.7:0/804126565 conn(0x7f70f00710d0 msgr2=0x7f70f00a4210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:00:20.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.781+0000 7f70f6f88700 1 -- 172.21.15.7:0/804126565 shutdown_connections 2023-09-30T02:00:20.782 INFO:teuthology.orchestra.run.smithi007.stderr:2023-09-30T02:00:20.781+0000 7f70f6f88700 1 -- 172.21.15.7:0/804126565 wait complete. 2023-09-30T02:00:20.793 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T02:00:20.794 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-30T02:00:20.794 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-30T02:00:20.794 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-09-30T02:00:20.795 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:00:20.835 INFO:tasks.ceph.osd.0:Stopped 2023-09-30T02:00:20.835 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-09-30T02:00:20.835 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:00:20.899 INFO:tasks.ceph.osd.1:Stopped 2023-09-30T02:00:20.899 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-09-30T02:00:20.899 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:00:20.963 INFO:tasks.ceph.osd.2:Stopped 2023-09-30T02:00:20.963 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-09-30T02:00:20.963 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:00:21.002 INFO:tasks.ceph.osd.3:Stopped 2023-09-30T02:00:21.002 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-09-30T02:00:21.003 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:00:21.082 INFO:tasks.ceph.osd.4:Stopped 2023-09-30T02:00:21.082 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-09-30T02:00:21.082 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:00:21.149 INFO:tasks.ceph.osd.5:Stopped 2023-09-30T02:00:21.149 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-30T02:00:21.150 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-09-30T02:00:21.150 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:00:21.218 INFO:tasks.ceph.mgr.x:Stopped 2023-09-30T02:00:21.218 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-30T02:00:21.218 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-09-30T02:00:21.218 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:00:21.229 INFO:tasks.ceph.mon.a:Stopped 2023-09-30T02:00:21.229 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-09-30T02:00:21.229 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:00:21.279 INFO:tasks.ceph.mon.c:Stopped 2023-09-30T02:00:21.280 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-09-30T02:00:21.280 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:00:21.292 INFO:tasks.ceph.mon.b:Stopped 2023-09-30T02:00:21.292 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-30T02:00:21.292 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(AUTH_BAD_CAPS\)' | head -n 1 2023-09-30T02:00:21.350 INFO:teuthology.orchestra.run.smithi007.stdout:1696038884.3782415 mon.a (mon.0) 81 : cluster [ERR] Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T02:00:21.350 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-09-30T02:00:21.351 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(AUTH_BAD_CAPS\)' | head -n 1 2023-09-30T02:00:21.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(AUTH_BAD_CAPS\)' | head -n 1 2023-09-30T02:00:21.490 INFO:teuthology.orchestra.run.smithi007.stdout:1696038884.3782415 mon.a (mon.0) 81 : cluster [ERR] Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T02:00:21.490 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi007.front.sepia.ceph.com 2023-09-30T02:00:21.491 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-09-30T02:00:21.770 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi007.front.sepia.ceph.com 2023-09-30T02:00:21.770 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-09-30T02:00:21.874 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi007.front.sepia.ceph.com 2023-09-30T02:00:21.874 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-09-30T02:00:21.954 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi050.front.sepia.ceph.com 2023-09-30T02:00:21.954 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-09-30T02:00:22.147 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi050.front.sepia.ceph.com 2023-09-30T02:00:22.147 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-09-30T02:00:22.230 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi050.front.sepia.ceph.com 2023-09-30T02:00:22.230 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-09-30T02:00:22.296 INFO:tasks.ceph:Archiving mon data... 2023-09-30T02:00:22.297 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407090/data/mon.a.tgz 2023-09-30T02:00:22.298 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2023-09-30T02:00:22.313 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.d9AAxqu1NE 2023-09-30T02:00:22.313 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.d9AAxqu1NE 2023-09-30T02:00:22.426 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.d9AAxqu1NE 2023-09-30T02:00:22.509 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.d9AAxqu1NE is 211KB 2023-09-30T02:00:22.602 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.d9AAxqu1NE 2023-09-30T02:00:22.617 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407090/data/mon.c.tgz 2023-09-30T02:00:22.617 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2023-09-30T02:00:22.670 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.zmWFhpjBJ8 2023-09-30T02:00:22.671 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.zmWFhpjBJ8 2023-09-30T02:00:22.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.zmWFhpjBJ8 2023-09-30T02:00:22.854 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.zmWFhpjBJ8 is 211KB 2023-09-30T02:00:22.947 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.zmWFhpjBJ8 2023-09-30T02:00:22.961 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407090/data/mon.b.tgz 2023-09-30T02:00:22.962 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2023-09-30T02:00:22.980 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.ey4UVxEavg 2023-09-30T02:00:22.980 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.ey4UVxEavg 2023-09-30T02:00:23.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.ey4UVxEavg 2023-09-30T02:00:23.180 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.ey4UVxEavg is 211KB 2023-09-30T02:00:23.272 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.ey4UVxEavg 2023-09-30T02:00:23.289 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-30T02:00:23.289 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T02:00:23.291 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T02:00:23.353 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-30T02:00:23.353 DEBUG:teuthology.orchestra.run.smithi007:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T02:00:23.358 DEBUG:teuthology.orchestra.run.smithi050:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T02:00:23.382 INFO:teuthology.orchestra.run.smithi007.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T02:00:23.419 INFO:teuthology.orchestra.run.smithi050.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T02:00:23.425 INFO:tasks.ceph:Archiving crash dumps... 2023-09-30T02:00:23.427 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407090/remote/smithi007/crash 2023-09-30T02:00:23.428 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T02:00:23.454 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407090/remote/smithi050/crash 2023-09-30T02:00:23.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T02:00:23.496 INFO:tasks.ceph:Compressing logs... 2023-09-30T02:00:23.496 DEBUG:teuthology.orchestra.run.smithi007:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T02:00:23.499 DEBUG:teuthology.orchestra.run.smithi050:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T02:00:24.282 INFO:tasks.ceph:Archiving logs... 2023-09-30T02:00:24.282 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407090/remote/smithi007/log 2023-09-30T02:00:24.283 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T02:00:24.511 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407090/remote/smithi050/log 2023-09-30T02:00:24.512 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T02:00:24.606 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T02:00:24.670 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=73d44d18fe6b4ebab2741f4802cd6ff2 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T02:00:24.673 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-30T02:00:24.693 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-30T02:00:24.736 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-30T02:00:24.736 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T02:00:24.748 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T02:00:24.821 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-30T02:00:24.829 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-30T02:00:24.830 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T02:00:24.832 DEBUG:teuthology.orchestra.run.smithi050:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T02:00:24.844 INFO:teuthology.orchestra.run.smithi007.stderr:bash: ntpq: command not found 2023-09-30T02:00:24.846 INFO:teuthology.orchestra.run.smithi050.stderr:bash: ntpq: command not found 2023-09-30T02:00:24.847 INFO:teuthology.orchestra.run.smithi007.stdout:210 Number of sources = 4 2023-09-30T02:00:24.847 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T02:00:24.847 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2023-09-30T02:00:24.848 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 2 -195us[ -195us] +/- 70ms 2023-09-30T02:00:24.848 INFO:teuthology.orchestra.run.smithi007.stdout:^* hv02.front.sepia.ceph.com 3 6 377 64 -117us[ -110us] +/- 55ms 2023-09-30T02:00:24.848 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 0 +497us[ +497us] +/- 89ms 2023-09-30T02:00:24.848 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T02:00:24.849 INFO:teuthology.orchestra.run.smithi050.stdout:210 Number of sources = 4 2023-09-30T02:00:24.850 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T02:00:24.850 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2023-09-30T02:00:24.850 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 64 -201us[ -169us] +/- 69ms 2023-09-30T02:00:24.850 INFO:teuthology.orchestra.run.smithi050.stdout:^* hv02.front.sepia.ceph.com 3 6 377 1 -81us[ -45us] +/- 56ms 2023-09-30T02:00:24.850 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 1 +514us[ +514us] +/- 89ms 2023-09-30T02:00:24.850 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T02:00:24.851 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-30T02:00:24.860 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-30T02:00:24.860 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-30T02:00:24.869 DEBUG:teuthology.orchestra.run.smithi007:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T02:00:24.985 DEBUG:teuthology.orchestra.run.smithi050:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T02:00:25.076 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T02:00:25.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T02:00:25.106 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T02:00:25.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T02:00:25.138 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-30T02:00:25.148 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-30T02:00:25.157 INFO:teuthology.task.internal:Duration was 1160.864795 seconds 2023-09-30T02:00:25.157 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-30T02:00:25.166 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-30T02:00:25.166 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T02:00:25.169 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T02:00:25.212 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T02:00:25.224 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T02:00:25.468 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-30T02:00:25.468 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi007.front.sepia.ceph.com 2023-09-30T02:00:25.468 DEBUG:teuthology.orchestra.run.smithi007:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T02:00:25.488 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi050.front.sepia.ceph.com 2023-09-30T02:00:25.488 DEBUG:teuthology.orchestra.run.smithi050:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T02:00:25.508 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-30T02:00:25.508 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T02:00:25.530 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T02:00:25.588 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-30T02:00:25.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T02:00:25.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T02:00:25.700 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-30T02:00:25.710 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-30T02:00:25.710 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T02:00:25.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T02:00:25.761 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-30T02:00:25.771 DEBUG:teuthology.orchestra.run.smithi007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T02:00:25.779 DEBUG:teuthology.orchestra.run.smithi050:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T02:00:25.801 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = core 2023-09-30T02:00:25.826 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = core 2023-09-30T02:00:25.840 DEBUG:teuthology.orchestra.run.smithi007:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T02:00:25.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T02:00:25.869 DEBUG:teuthology.orchestra.run.smithi050:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T02:00:25.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T02:00:25.895 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-30T02:00:25.904 INFO:teuthology.task.internal:Transferring archived files... 2023-09-30T02:00:25.905 DEBUG:teuthology.misc:Transferring archived files from smithi007:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407090/remote/smithi007 2023-09-30T02:00:25.905 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T02:00:25.945 DEBUG:teuthology.misc:Transferring archived files from smithi050:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407090/remote/smithi050 2023-09-30T02:00:25.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T02:00:25.981 INFO:teuthology.task.internal:Removing archive directory... 2023-09-30T02:00:25.982 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T02:00:25.984 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T02:00:26.031 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-30T02:00:26.042 INFO:teuthology.task.internal:Not uploading archives. 2023-09-30T02:00:26.042 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-30T02:00:26.052 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-30T02:00:26.052 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T02:00:26.054 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T02:00:26.067 INFO:teuthology.orchestra.run.smithi007.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 02:00 /home/ubuntu/cephtest 2023-09-30T02:00:26.085 INFO:teuthology.orchestra.run.smithi050.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 02:00 /home/ubuntu/cephtest 2023-09-30T02:00:26.086 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-30T02:00:26.097 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-30T02:00:26.118 INFO:teuthology.nuke:Checking targets against current locks 2023-09-30T02:00:26.146 DEBUG:teuthology.nuke:shortname: smithi007 2023-09-30T02:00:26.146 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T02:00:26.171 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407090', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 01:33:51.626510', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL4Ne8rnE0fKgnLxndlIRGawJ6kI6jVZPJd3ia5X2z5siGW7NsynibbkY3Sa/LPpAgVwpTy1xB+DLeEy9osBQQA='} 2023-09-30T02:00:26.175 DEBUG:teuthology.nuke:shortname: smithi050 2023-09-30T02:00:26.175 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T02:00:26.200 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407090', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 01:33:51.628404', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPKVvvKVeE2oRIix/AIkKakF3O4QIvkSEqHCqfZMbbIAoWcstU9WTGbXpT9G0dSp1ejGKy6wHwNz/gMXjOyAK9U='} 2023-09-30T02:00:26.226 INFO:teuthology.orchestra.console.smithi007:Power off 2023-09-30T02:00:26.227 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T02:00:26.249 DEBUG:teuthology.orchestra.console.smithi007:power off output: Chassis Power Control: Down/Off 2023-09-30T02:00:26.250 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:00:26.269 INFO:teuthology.orchestra.console.smithi050:Power off 2023-09-30T02:00:26.270 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T02:00:26.279 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2023-09-30T02:00:26.293 DEBUG:teuthology.orchestra.console.smithi050:power off output: Chassis Power Control: Down/Off 2023-09-30T02:00:26.293 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:00:26.316 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2023-09-30T02:00:30.282 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:00:30.317 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:00:30.392 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2023-09-30T02:00:30.429 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2023-09-30T02:00:34.394 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:00:34.430 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:00:34.504 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is off 2023-09-30T02:00:34.541 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is off 2023-09-30T02:00:34.606 INFO:teuthology.orchestra.console.smithi007:Power off completed 2023-09-30T02:00:34.642 INFO:teuthology.orchestra.console.smithi050:Power off completed 2023-09-30T02:00:34.747 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/one workloads/rados_mon_workunits} duration: 1160.8647949695587 failure_reason: '"1696038884.3782415 mon.a (mon.0) 81 : cluster [ERR] Health check failed: Module ''pg_autoscaler'' has failed: ''PgAutoscaler'' object has no attribute ''noautoscale'' (MGR_MODULE_ERROR)" in cluster log' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=73d44d18fe6b4ebab2741f4802cd6ff2 status: fail success: false 2023-09-30T02:00:34.747 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T02:00:34.836 INFO:teuthology.run:FAIL